Jul 23 11:33:41.946054 ip-10-0-141-229 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Jul 23 11:33:41.946069 ip-10-0-141-229 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Jul 23 11:33:41.946079 ip-10-0-141-229 systemd[1]: kubelet.service: Failed with result 'resources'.
Jul 23 11:33:41.946405 ip-10-0-141-229 systemd[1]: Failed to start Kubernetes Kubelet.
Jul 23 11:33:52.151075 ip-10-0-141-229 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Jul 23 11:33:52.151096 ip-10-0-141-229 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot da15e50119da4b05982ec59fa22dd730 --
Jul 23 11:35:45.495973 ip-10-0-141-229 systemd[1]: Starting Kubernetes Kubelet...
Jul 23 11:35:45.853235 ip-10-0-141-229 kubenswrapper[2563]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 23 11:35:45.853235 ip-10-0-141-229 kubenswrapper[2563]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 23 11:35:45.853235 ip-10-0-141-229 kubenswrapper[2563]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 23 11:35:45.853235 ip-10-0-141-229 kubenswrapper[2563]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 23 11:35:45.853235 ip-10-0-141-229 kubenswrapper[2563]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 23 11:35:45.855961 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.855870    2563 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 23 11:35:45.861567 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.861551    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 23 11:35:45.861567 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.861566    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 23 11:35:45.861625 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.861570    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 23 11:35:45.861625 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.861573    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 23 11:35:45.861625 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.861576    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 23 11:35:45.861625 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.861579    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 23 11:35:45.861625 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.861582    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 23 11:35:45.861625 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.861585    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 23 11:35:45.861625 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.861588    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 23 11:35:45.861625 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.861591    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 23 11:35:45.861625 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.861593    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 23 11:35:45.861625 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.861596    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 23 11:35:45.861625 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.861598    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 23 11:35:45.861625 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.861601    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 23 11:35:45.861625 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.861604    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 23 11:35:45.861625 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.861606    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 23 11:35:45.861625 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.861609    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 23 11:35:45.861625 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.861611    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 23 11:35:45.861625 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.861614    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 23 11:35:45.861625 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.861616    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 23 11:35:45.861625 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.861619    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 23 11:35:45.861625 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.861621    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 23 11:35:45.862114 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.861624    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 23 11:35:45.862114 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.861626    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 23 11:35:45.862114 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.861629    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 23 11:35:45.862114 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.861631    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 23 11:35:45.862114 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.861634    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 23 11:35:45.862114 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.861637    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 23 11:35:45.862114 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.861640    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 23 11:35:45.862114 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.861644    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 23 11:35:45.862114 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.861648    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 23 11:35:45.862114 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.861652    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 23 11:35:45.862114 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.861655    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 23 11:35:45.862114 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.861657    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 23 11:35:45.862114 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.861660    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 23 11:35:45.862114 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.861663    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 23 11:35:45.862114 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.861665    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 23 11:35:45.862114 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.861668    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 23 11:35:45.862114 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.861671    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 23 11:35:45.862114 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.861673    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 23 11:35:45.862114 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.861675    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 23 11:35:45.862605 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.861679    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 23 11:35:45.862605 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.861699    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 23 11:35:45.862605 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.861715    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 23 11:35:45.862605 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.861717    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 23 11:35:45.862605 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.861720    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 23 11:35:45.862605 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.861723    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 23 11:35:45.862605 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.861725    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 23 11:35:45.862605 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.861728    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 23 11:35:45.862605 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.861730    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 23 11:35:45.862605 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.861732    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 23 11:35:45.862605 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.861735    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 23 11:35:45.862605 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.861737    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 23 11:35:45.862605 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.861739    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 23 11:35:45.862605 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.861742    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 23 11:35:45.862605 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.861746    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 23 11:35:45.862605 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.861749    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 23 11:35:45.862605 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.861751    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 23 11:35:45.862605 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.861754    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 23 11:35:45.862605 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.861757    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 23 11:35:45.862605 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.861759    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 23 11:35:45.863111 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.861762    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 23 11:35:45.863111 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.861765    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 23 11:35:45.863111 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.861767    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 23 11:35:45.863111 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.861770    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 23 11:35:45.863111 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.861773    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 23 11:35:45.863111 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.861776    2563 feature_gate.go:328] unrecognized feature gate: Example
Jul 23 11:35:45.863111 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.861778    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 23 11:35:45.863111 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.861780    2563 feature_gate.go:328] unrecognized feature gate: Example2
Jul 23 11:35:45.863111 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.861783    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 23 11:35:45.863111 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.861785    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 23 11:35:45.863111 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.861788    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 23 11:35:45.863111 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.861790    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 23 11:35:45.863111 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.861793    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 23 11:35:45.863111 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.861795    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 23 11:35:45.863111 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.861798    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 23 11:35:45.863111 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.861800    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 23 11:35:45.863111 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.861803    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 23 11:35:45.863111 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.861805    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 23 11:35:45.863111 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.861807    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 23 11:35:45.863111 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.861810    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 23 11:35:45.863612 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.861813    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 23 11:35:45.863612 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.861817    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 23 11:35:45.863612 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.861820    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 23 11:35:45.863612 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.861822    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 23 11:35:45.863612 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.861825    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 23 11:35:45.863612 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.861827    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 23 11:35:45.863612 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.861831    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 23 11:35:45.863612 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.861835    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 23 11:35:45.863612 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.861838    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 23 11:35:45.863612 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.861842    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 23 11:35:45.863612 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.861846    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 23 11:35:45.863612 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.861849    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 23 11:35:45.863612 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.861852    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 23 11:35:45.863612 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.861854    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 23 11:35:45.863612 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.861857    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 23 11:35:45.863612 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.861860    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 23 11:35:45.863612 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.861863    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 23 11:35:45.863612 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.861866    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 23 11:35:45.863612 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.862554    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 23 11:35:45.864088 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.862561    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 23 11:35:45.864088 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.862564    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 23 11:35:45.864088 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.862567    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 23 11:35:45.864088 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.862569    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 23 11:35:45.864088 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.862571    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 23 11:35:45.864088 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.862574    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 23 11:35:45.864088 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.862576    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 23 11:35:45.864088 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.862579    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 23 11:35:45.864088 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.862581    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 23 11:35:45.864088 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.862584    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 23 11:35:45.864088 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.862589    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 23 11:35:45.864088 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.862593    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 23 11:35:45.864088 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.862596    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 23 11:35:45.864088 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.862598    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 23 11:35:45.864088 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.862601    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 23 11:35:45.864088 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.862604    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 23 11:35:45.864088 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.862608    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 23 11:35:45.864088 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.862610    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 23 11:35:45.864088 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.862614    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 23 11:35:45.864540 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.862618    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 23 11:35:45.864540 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.862621    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 23 11:35:45.864540 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.862625    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 23 11:35:45.864540 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.862628    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 23 11:35:45.864540 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.862630    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 23 11:35:45.864540 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.862633    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 23 11:35:45.864540 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.862636    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 23 11:35:45.864540 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.862639    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 23 11:35:45.864540 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.862642    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 23 11:35:45.864540 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.862644    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 23 11:35:45.864540 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.862647    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 23 11:35:45.864540 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.862650    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 23 11:35:45.864540 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.862653    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 23 11:35:45.864540 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.862655    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 23 11:35:45.864540 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.862657    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 23 11:35:45.864540 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.862660    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 23 11:35:45.864540 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.862663    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 23 11:35:45.864540 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.862667    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 23 11:35:45.864540 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.862671    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 23 11:35:45.864540 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.862674    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 23 11:35:45.865077 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.862676    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 23 11:35:45.865077 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.862679    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 23 11:35:45.865077 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.862697    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 23 11:35:45.865077 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.862700    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 23 11:35:45.865077 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.862703    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 23 11:35:45.865077 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.862706    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 23 11:35:45.865077 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.862708    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 23 11:35:45.865077 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.862711    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 23 11:35:45.865077 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.862714    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 23 11:35:45.865077 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.862717    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 23 11:35:45.865077 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.862719    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 23 11:35:45.865077 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.862721    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 23 11:35:45.865077 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.862724    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 23 11:35:45.865077 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.862727    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 23 11:35:45.865077 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.862730    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 23 11:35:45.865077 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.862732    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 23 11:35:45.865077 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.862735    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 23 11:35:45.865077 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.862737    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 23 11:35:45.865077 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.862739    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 23 11:35:45.865077 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.862742    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 23 11:35:45.865557 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.862744    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 23 11:35:45.865557 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.862747    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 23 11:35:45.865557 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.862749    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 23 11:35:45.865557 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.862751    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 23 11:35:45.865557 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.862754    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 23 11:35:45.865557 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.862757    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 23 11:35:45.865557 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.862759    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 23 11:35:45.865557 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.862762    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 23 11:35:45.865557 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.862764    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 23 11:35:45.865557 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.862766    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 23 11:35:45.865557 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.862769    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 23 11:35:45.865557 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.862771    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 23 11:35:45.865557 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.862774    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 23 11:35:45.865557 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.862776    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 23 11:35:45.865557 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.862779    2563 feature_gate.go:328] unrecognized feature gate: Example2
Jul 23 11:35:45.865557 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.862782    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 23 11:35:45.865557 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.862785    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 23 11:35:45.865557 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.862788    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 23 11:35:45.865557 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.862790    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 23 11:35:45.865557 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.862793    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 23 11:35:45.866044 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.862795    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 23 11:35:45.866044 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.862797    2563 feature_gate.go:328] unrecognized feature gate: Example
Jul 23 11:35:45.866044 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.862800    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 23 11:35:45.866044 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.862803    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 23 11:35:45.866044 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.862805    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 23 11:35:45.866044 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.862808    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 23 11:35:45.866044 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.862810    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 23 11:35:45.866044 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.862813    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 23 11:35:45.866044 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.862815    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 23 11:35:45.866044 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.862817    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 23 11:35:45.866044 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.862820    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 23 11:35:45.866044 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.862823    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 23 11:35:45.866044 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.862825    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 23 11:35:45.866044 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.862827    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 23 11:35:45.866044 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.862830    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 23 11:35:45.866044 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.862832    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 23 11:35:45.866044 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.862834    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 23 11:35:45.866044 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.862837    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 23 11:35:45.866044 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.862840    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 23 11:35:45.866044 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.863213    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 23 11:35:45.866527 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.863218    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 23 11:35:45.866527 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.863220    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 23 11:35:45.866527 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.863223    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 23 11:35:45.866527 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.863226    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 23 11:35:45.866527 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.863228    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 23 11:35:45.866527 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.863231    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 23 11:35:45.866527 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.863233    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 23 11:35:45.866527 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.863236    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 23 11:35:45.866527 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.863241    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 23 11:35:45.866527 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.863243    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 23 11:35:45.866527 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.863246    2563 feature_gate.go:328] unrecognized feature gate: Example2
Jul 23 11:35:45.866527 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.863248    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 23 11:35:45.866527 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.863251    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 23 11:35:45.866527 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.863253    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 23 11:35:45.866527 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.863256    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 23 11:35:45.866527 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.863258    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 23 11:35:45.866527 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.863261    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 23 11:35:45.866527 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.863265    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 23 11:35:45.866527 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.863268    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 23 11:35:45.866527 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.863270    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 23 11:35:45.867026 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.863273    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 23 11:35:45.867026 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.863275    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 23 11:35:45.867026 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.863278    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 23 11:35:45.867026 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.863280    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 23 11:35:45.867026 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.863282    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 23 11:35:45.867026 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.863285    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 23 11:35:45.867026 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.863287    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 23 11:35:45.867026 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.863290    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 23 11:35:45.867026 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.863292    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 23 11:35:45.867026 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.863295    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 23 11:35:45.867026 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.863297    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 23 11:35:45.867026 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.863300    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 23 11:35:45.867026 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.863302    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 23 11:35:45.867026 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.863304    2563 feature_gate.go:328] unrecognized feature gate: Example
Jul 23 11:35:45.867026 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.863307    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 23 11:35:45.867026 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.863309    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 23 11:35:45.867026 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.863312    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 23 11:35:45.867026 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.863314    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 23 11:35:45.867026 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.863317    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 23 11:35:45.867026 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.863319    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 23 11:35:45.867515 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.863322    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 23 11:35:45.867515 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.863327    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 23 11:35:45.867515 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.863330    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 23 11:35:45.867515 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.863333    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 23 11:35:45.867515 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.863336    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 23 11:35:45.867515 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.863339    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 23 11:35:45.867515 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.863342    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 23 11:35:45.867515 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.863345    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 23 11:35:45.867515 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.863348    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 23 11:35:45.867515 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.863353    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 23 11:35:45.867515 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.863355    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 23 11:35:45.867515 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.863358    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 23 11:35:45.867515 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.863361    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 23 11:35:45.867515 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.863363    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 23 11:35:45.867515 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.863366    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 23 11:35:45.867515 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.863368    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 23 11:35:45.867515 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.863371    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 23 11:35:45.867515 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.863374    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 23 11:35:45.867515 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.863376    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 23 11:35:45.867989 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.863378    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 23 11:35:45.867989 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.863382    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 23 11:35:45.867989 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.863384    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 23 11:35:45.867989 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.863387    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 23 11:35:45.867989 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.863390    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 23 11:35:45.867989 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.863392    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 23 11:35:45.867989 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.863395    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 23 11:35:45.867989 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.863397    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 23 11:35:45.867989 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.863399    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 23 11:35:45.867989 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.863402    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 23 11:35:45.867989 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.863404    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 23 11:35:45.867989 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.863407    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 23 11:35:45.867989 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.863409    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 23 11:35:45.867989 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.863413    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 23 11:35:45.867989 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.863415    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 23 11:35:45.867989 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.863418    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 23 11:35:45.867989 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.863420    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 23 11:35:45.867989 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.863423    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 23 11:35:45.867989 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.863425    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 23 11:35:45.868582 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.863428    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 23 11:35:45.868582 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.863430    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 23 11:35:45.868582 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.863433    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 23 11:35:45.868582 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.863437    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 23 11:35:45.868582 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.863439    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 23 11:35:45.868582 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.863441    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 23 11:35:45.868582 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.863444    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 23 11:35:45.868582 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.863447    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 23 11:35:45.868582 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.863449    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 23 11:35:45.868582 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.863451    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 23 11:35:45.868582 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.863454    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 23 11:35:45.868582 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.863456    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 23 11:35:45.868582 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.863458    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 23 11:35:45.868582 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.863461    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 23 11:35:45.868582 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.863465    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 23 11:35:45.868582 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.863468    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 23 11:35:45.868582 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.863471    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 23 11:35:45.868582 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.863475    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 23 11:35:45.868582 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.863477    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 23 11:35:45.868582 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.863480    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 23 11:35:45.869126 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.863548    2563 flags.go:64] FLAG: --address="0.0.0.0"
Jul 23 11:35:45.869126 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.863555    2563 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 23 11:35:45.869126 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.863561    2563 flags.go:64] FLAG: --anonymous-auth="true"
Jul 23 11:35:45.869126 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.863565    2563 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 23 11:35:45.869126 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.863570    2563 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 23 11:35:45.869126 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.863573    2563 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 23 11:35:45.869126 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.863578    2563 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 23 11:35:45.869126 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.863582    2563 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 23 11:35:45.869126 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.863585    2563 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 23 11:35:45.869126 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.863588    2563 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 23 11:35:45.869126 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.863591    2563 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 23 11:35:45.869126 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.863594    2563 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 23 11:35:45.869126 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.863597    2563 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 23 11:35:45.869126 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.863600    2563 flags.go:64] FLAG: --cgroup-root=""
Jul 23 11:35:45.869126 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.863603    2563 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 23 11:35:45.869126 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.863608    2563 flags.go:64] FLAG: --client-ca-file=""
Jul 23 11:35:45.869126 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.863611    2563 flags.go:64] FLAG: --cloud-provider="external"
Jul 23 11:35:45.869126 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.863614    2563 flags.go:64] FLAG: --cluster-dns="[]"
Jul 23 11:35:45.869126 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.863618    2563 flags.go:64] FLAG: --cluster-domain=""
Jul 23 11:35:45.869126 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.863621    2563 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 23 11:35:45.869126 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.863624    2563 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 23 11:35:45.869126 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.863627    2563 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 23 11:35:45.869126 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.863630    2563 flags.go:64] FLAG: --container-log-max-files="5"
Jul 23 11:35:45.869126 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.863634    2563 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 23 11:35:45.869715 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.863637    2563 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 23 11:35:45.869715 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.863640    2563 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 23 11:35:45.869715 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.863643    2563 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 23 11:35:45.869715 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.863646    2563 flags.go:64] FLAG: --contention-profiling="false"
Jul 23 11:35:45.869715 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.863648    2563 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 23 11:35:45.869715 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.863651    2563 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 23 11:35:45.869715 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.863655    2563 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 23 11:35:45.869715 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.863657    2563 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 23 11:35:45.869715 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.863661    2563 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 23 11:35:45.869715 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.863664    2563 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 23 11:35:45.869715 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.863667    2563 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 23 11:35:45.869715 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.863669    2563 flags.go:64] FLAG: --enable-load-reader="false"
Jul 23 11:35:45.869715 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.863672    2563 flags.go:64] FLAG: --enable-server="true"
Jul 23 11:35:45.869715 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.863675    2563 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 23 11:35:45.869715 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.863696    2563 flags.go:64] FLAG: --event-burst="100"
Jul 23 11:35:45.869715 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.863700    2563 flags.go:64] FLAG: --event-qps="50"
Jul 23 11:35:45.869715 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.863702    2563 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 23 11:35:45.869715 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.863706    2563 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 23 11:35:45.869715 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.863709    2563 flags.go:64] FLAG: --eviction-hard=""
Jul 23 11:35:45.869715 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.863712    2563 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 23 11:35:45.869715 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.863715    2563 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 23 11:35:45.869715 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.863718    2563 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 23 11:35:45.869715 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.863721    2563 flags.go:64] FLAG: --eviction-soft=""
Jul 23 11:35:45.869715 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.863725    2563 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 23 11:35:45.869715 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.863727    2563 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 23 11:35:45.870362 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.863730    2563 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 23 11:35:45.870362 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.863733    2563 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 23 11:35:45.870362 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.863736    2563 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 23 11:35:45.870362 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.863739    2563 flags.go:64] FLAG: --fail-swap-on="true"
Jul 23 11:35:45.870362 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.863741    2563 flags.go:64] FLAG: --feature-gates=""
Jul 23 11:35:45.870362 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.863745    2563 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 23 11:35:45.870362 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.863748    2563 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 23 11:35:45.870362 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.863751    2563 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 23 11:35:45.870362 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.863754    2563 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 23 11:35:45.870362 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.863757    2563 flags.go:64] FLAG: --healthz-port="10248"
Jul 23 11:35:45.870362 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.863760    2563 flags.go:64] FLAG: --help="false"
Jul 23 11:35:45.870362 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.863762    2563 flags.go:64] FLAG: --hostname-override="ip-10-0-141-229.ec2.internal"
Jul 23 11:35:45.870362 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.863765    2563 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 23 11:35:45.870362 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.863768    2563 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 23 11:35:45.870362 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.863772    2563 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 23 11:35:45.870362 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.863775    2563 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 23 11:35:45.870362 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.863778    2563 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 23 11:35:45.870362 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.863781    2563 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 23 11:35:45.870362 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.863783    2563 flags.go:64] FLAG: --image-service-endpoint=""
Jul 23 11:35:45.870362 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.863786    2563 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 23 11:35:45.870362 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.863789    2563 flags.go:64] FLAG: --kube-api-burst="100"
Jul 23 11:35:45.870362 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.863793    2563 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 23 11:35:45.870362 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.863796    2563 flags.go:64] FLAG: --kube-api-qps="50"
Jul 23 11:35:45.870362 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.863799    2563 flags.go:64] FLAG: --kube-reserved=""
Jul 23 11:35:45.871022 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.863802    2563 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 23 11:35:45.871022 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.863804    2563 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 23 11:35:45.871022 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.863807    2563 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 23 11:35:45.871022 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.863810    2563 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 23 11:35:45.871022 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.863812    2563 flags.go:64] FLAG: --lock-file=""
Jul 23 11:35:45.871022 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.863815    2563 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 23 11:35:45.871022 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.863818    2563 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 23 11:35:45.871022 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.863821    2563 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 23 11:35:45.871022 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.863827    2563 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 23 11:35:45.871022 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.863829    2563 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 23 11:35:45.871022 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.863832    2563 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 23 11:35:45.871022 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.863835    2563 flags.go:64] FLAG: --logging-format="text"
Jul 23 11:35:45.871022 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.863837    2563 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 23 11:35:45.871022 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.863840    2563 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 23 11:35:45.871022 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.863843    2563 flags.go:64] FLAG: --manifest-url=""
Jul 23 11:35:45.871022 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.863845    2563 flags.go:64] FLAG: --manifest-url-header=""
Jul 23 11:35:45.871022 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.863849    2563 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 23 11:35:45.871022 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.863852    2563 flags.go:64] FLAG: --max-open-files="1000000"
Jul 23 11:35:45.871022 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.863856    2563 flags.go:64] FLAG: --max-pods="110"
Jul 23 11:35:45.871022 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.863859    2563 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 23 11:35:45.871022 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.863862    2563 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 23 11:35:45.871022 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.863865    2563 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 23 11:35:45.871022 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.863867    2563 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 23 11:35:45.871022 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.863870    2563 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 23 11:35:45.871022 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.863873    2563 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 23 11:35:45.871631 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.863876    2563 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 23 11:35:45.871631 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.863883    2563 flags.go:64] FLAG: --node-status-max-images="50"
Jul 23 11:35:45.871631 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.863886    2563 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 23 11:35:45.871631 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.863889    2563 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 23 11:35:45.871631 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.863893    2563 flags.go:64] FLAG: --pod-cidr=""
Jul 23 11:35:45.871631 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.863896    2563 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4017e99d76d7b91598d3a3430cd0eba6eda8b33653addd3838fdfe3cca69c5a0"
Jul 23 11:35:45.871631 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.863901    2563 flags.go:64] FLAG: --pod-manifest-path=""
Jul 23 11:35:45.871631 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.863904    2563 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 23 11:35:45.871631 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.863907    2563 flags.go:64] FLAG: --pods-per-core="0"
Jul 23 11:35:45.871631 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.863910    2563 flags.go:64] FLAG: --port="10250"
Jul 23 11:35:45.871631 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.863913    2563 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 23 11:35:45.871631 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.863915    2563 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-091b9ec5e9b28d931"
Jul 23 11:35:45.871631 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.863918    2563 flags.go:64] FLAG: --qos-reserved=""
Jul 23 11:35:45.871631 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.863921    2563 flags.go:64] FLAG: --read-only-port="10255"
Jul 23 11:35:45.871631 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.863924    2563 flags.go:64] FLAG: --register-node="true"
Jul 23 11:35:45.871631 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.863927    2563 flags.go:64] FLAG: --register-with-taints=""
Jul 23 11:35:45.871631 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.863931    2563 flags.go:64] FLAG: --registry-burst="10"
Jul 23 11:35:45.871631 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.863934    2563 flags.go:64] FLAG: --registry-qps="5"
Jul 23 11:35:45.871631 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.863937    2563 flags.go:64] FLAG: --reserved-cpus=""
Jul 23 11:35:45.871631 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.863939    2563 flags.go:64] FLAG: --reserved-memory=""
Jul 23 11:35:45.871631 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.863943    2563 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 23 11:35:45.871631 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.863945    2563 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 23 11:35:45.871631 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.863948    2563 flags.go:64] FLAG: --rotate-certificates="false"
Jul 23 11:35:45.871631 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.863951    2563 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 23 11:35:45.871631 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.863954    2563 flags.go:64] FLAG: --runonce="false"
Jul 23 11:35:45.872273 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.863956    2563 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 23 11:35:45.872273 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.863959    2563 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 23 11:35:45.872273 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.863962    2563 flags.go:64] FLAG: --seccomp-default="false"
Jul 23 11:35:45.872273 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.863965    2563 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 23 11:35:45.872273 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.863968    2563 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 23 11:35:45.872273 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.863971    2563 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 23 11:35:45.872273 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.863974    2563 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 23 11:35:45.872273 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.863977    2563 flags.go:64] FLAG: --storage-driver-password="root"
Jul 23 11:35:45.872273 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.863980    2563 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 23 11:35:45.872273 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.863982    2563 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 23 11:35:45.872273 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.863985    2563 flags.go:64] FLAG: --storage-driver-user="root"
Jul 23 11:35:45.872273 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.863988    2563 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 23 11:35:45.872273 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.863992    2563 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 23 11:35:45.872273 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.863994    2563 flags.go:64] FLAG: --system-cgroups=""
Jul 23 11:35:45.872273 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.863997    2563 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 23 11:35:45.872273 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.864002    2563 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 23 11:35:45.872273 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.864005    2563 flags.go:64] FLAG: --tls-cert-file=""
Jul 23 11:35:45.872273 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.864007    2563 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 23 11:35:45.872273 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.864011    2563 flags.go:64] FLAG: --tls-min-version=""
Jul 23 11:35:45.872273 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.864014    2563 flags.go:64] FLAG: --tls-private-key-file=""
Jul 23 11:35:45.872273 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.864017    2563 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 23 11:35:45.872273 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.864019    2563 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 23 11:35:45.872273 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.864022    2563 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 23 11:35:45.872273 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.864025    2563 flags.go:64] FLAG: --v="2"
Jul 23 11:35:45.872273 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.864029    2563 flags.go:64] FLAG: --version="false"
Jul 23 11:35:45.872981 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.864032    2563 flags.go:64] FLAG: --vmodule=""
Jul 23 11:35:45.872981 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.864036    2563 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 23 11:35:45.872981 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.864039    2563 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 23 11:35:45.872981 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.864146    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 23 11:35:45.872981 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.864150    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 23 11:35:45.872981 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.864153    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 23 11:35:45.872981 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.864156    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 23 11:35:45.872981 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.864158    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 23 11:35:45.872981 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.864161    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 23 11:35:45.872981 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.864164    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 23 11:35:45.872981 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.864166    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 23 11:35:45.872981 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.864169    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 23 11:35:45.872981 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.864171    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 23 11:35:45.872981 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.864173    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 23 11:35:45.872981 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.864176    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 23 11:35:45.872981 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.864179    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 23 11:35:45.872981 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.864181    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 23 11:35:45.872981 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.864184    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 23 11:35:45.872981 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.864186    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 23 11:35:45.872981 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.864189    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 23 11:35:45.872981 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.864192    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 23 11:35:45.873841 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.864195    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 23 11:35:45.873841 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.864197    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 23 11:35:45.873841 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.864199    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 23 11:35:45.873841 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.864202    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 23 11:35:45.873841 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.864204    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 23 11:35:45.873841 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.864207    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 23 11:35:45.873841 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.864211    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 23 11:35:45.873841 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.864213    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 23 11:35:45.873841 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.864216    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 23 11:35:45.873841 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.864219    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 23 11:35:45.873841 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.864223    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 23 11:35:45.873841 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.864225    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 23 11:35:45.873841 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.864228    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 23 11:35:45.873841 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.864231    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 23 11:35:45.873841 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.864233    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 23 11:35:45.873841 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.864235    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 23 11:35:45.873841 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.864238    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 23 11:35:45.873841 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.864240    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 23 11:35:45.873841 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.864243    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 23 11:35:45.874631 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.864245    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 23 11:35:45.874631 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.864247    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 23 11:35:45.874631 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.864250    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 23 11:35:45.874631 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.864253    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 23 11:35:45.874631 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.864255    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 23 11:35:45.874631 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.864258    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 23 11:35:45.874631 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.864261    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 23 11:35:45.874631 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.864264    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 23 11:35:45.874631 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.864267    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 23 11:35:45.874631 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.864269    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 23 11:35:45.874631 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.864272    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 23 11:35:45.874631 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.864274    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 23 11:35:45.874631 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.864277    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 23 11:35:45.874631 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.864279    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 23 11:35:45.874631 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.864281    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 23 11:35:45.874631 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.864284    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 23 11:35:45.874631 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.864286    2563 feature_gate.go:328] unrecognized feature gate: Example2
Jul 23 11:35:45.874631 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.864289    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 23 11:35:45.874631 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.864291    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 23 11:35:45.874631 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.864295    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 23 11:35:45.875341 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.864298    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 23 11:35:45.875341 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.864300    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 23 11:35:45.875341 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.864303    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 23 11:35:45.875341 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.864305    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 23 11:35:45.875341 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.864308    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 23 11:35:45.875341 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.864310    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 23 11:35:45.875341 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.864313    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 23 11:35:45.875341 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.864315    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 23 11:35:45.875341 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.864319    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 23 11:35:45.875341 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.864323    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 23 11:35:45.875341 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.864326    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 23 11:35:45.875341 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.864328    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 23 11:35:45.875341 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.864331    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 23 11:35:45.875341 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.864333    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 23 11:35:45.875341 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.864336    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 23 11:35:45.875341 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.864339    2563 feature_gate.go:328] unrecognized feature gate: Example
Jul 23 11:35:45.875341 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.864341    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 23 11:35:45.875341 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.864344    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 23 11:35:45.875341 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.864346    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 23 11:35:45.876109 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.864349    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 23 11:35:45.876109 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.864352    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 23 11:35:45.876109 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.864354    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 23 11:35:45.876109 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.864357    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 23 11:35:45.876109 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.864359    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 23 11:35:45.876109 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.864362    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 23 11:35:45.876109 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.864364    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 23 11:35:45.876109 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.864366    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 23 11:35:45.876109 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.864368    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 23 11:35:45.876109 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.864371    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 23 11:35:45.876109 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.864374    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 23 11:35:45.876109 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.864377    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 23 11:35:45.876109 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.864381    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 23 11:35:45.876109 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.864383    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 23 11:35:45.876109 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.864386    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 23 11:35:45.876109 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.864388    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 23 11:35:45.876109 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.864390    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 23 11:35:45.876109 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.864393    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 23 11:35:45.876109 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.864395    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 23 11:35:45.876904 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.864397    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 23 11:35:45.876904 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.864400    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 23 11:35:45.876904 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.864402    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 23 11:35:45.876904 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.864405    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 23 11:35:45.876904 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.865480    2563 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 23 11:35:45.876904 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.872150    2563 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 23 11:35:45.876904 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.872163    2563 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 23 11:35:45.876904 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.872182    2563 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 23 11:35:45.876904 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.872187    2563 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 23 11:35:45.876904 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.872492    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 23 11:35:45.876904 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.872508    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 23 11:35:45.876904 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.872516    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 23 11:35:45.876904 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.872524    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 23 11:35:45.876904 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.872533    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 23 11:35:45.877265 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.872543    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 23 11:35:45.877265 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.872550    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 23 11:35:45.877265 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.872557    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 23 11:35:45.877265 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.872564    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 23 11:35:45.877265 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.872579    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 23 11:35:45.877265 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.872587    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 23 11:35:45.877265 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.872594    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 23 11:35:45.877265 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.872603    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 23 11:35:45.877265 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.872611    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 23 11:35:45.877265 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.872619    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 23 11:35:45.877265 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.872626    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 23 11:35:45.877265 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.872634    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 23 11:35:45.877265 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.872642    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 23 11:35:45.877265 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.872650    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 23 11:35:45.877265 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.872658    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 23 11:35:45.877265 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.872666    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 23 11:35:45.877265 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.872678    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 23 11:35:45.877265 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.872703    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 23 11:35:45.877265 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.872710    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 23 11:35:45.877265 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.872718    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 23 11:35:45.877907 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.872724    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 23 11:35:45.877907 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.872731    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 23 11:35:45.877907 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.872740    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 23 11:35:45.877907 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.872747    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 23 11:35:45.877907 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.872755    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 23 11:35:45.877907 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.872762    2563 feature_gate.go:328] unrecognized feature gate: Example
Jul 23 11:35:45.877907 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.872768    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 23 11:35:45.877907 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.872776    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 23 11:35:45.877907 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.872783    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 23 11:35:45.877907 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.872796    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 23 11:35:45.877907 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.872803    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 23 11:35:45.877907 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.872810    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 23 11:35:45.877907 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.872817    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 23 11:35:45.877907 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.872826    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 23 11:35:45.877907 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.872834    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 23 11:35:45.877907 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.872841    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 23 11:35:45.877907 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.872850    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 23 11:35:45.877907 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.872859    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 23 11:35:45.877907 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.872867    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 23 11:35:45.878402 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.872875    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 23 11:35:45.878402 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.872882    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 23 11:35:45.878402 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.872891    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 23 11:35:45.878402 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.872895    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 23 11:35:45.878402 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.872900    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 23 11:35:45.878402 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.872906    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 23 11:35:45.878402 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.872909    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 23 11:35:45.878402 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.872914    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 23 11:35:45.878402 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.872918    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 23 11:35:45.878402 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.872922    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 23 11:35:45.878402 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.872926    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 23 11:35:45.878402 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.872930    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 23 11:35:45.878402 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.872934    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 23 11:35:45.878402 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.872937    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 23 11:35:45.878402 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.872942    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 23 11:35:45.878402 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.872950    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 23 11:35:45.878402 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.872954    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 23 11:35:45.878402 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.872959    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 23 11:35:45.878402 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.872963    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 23 11:35:45.878888 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.872967    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 23 11:35:45.878888 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.872971    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 23 11:35:45.878888 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.872977    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 23 11:35:45.878888 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.872982    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 23 11:35:45.878888 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.872987    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 23 11:35:45.878888 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.872992    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 23 11:35:45.878888 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.872996    2563 feature_gate.go:328] unrecognized feature gate: Example2
Jul 23 11:35:45.878888 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.873000    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 23 11:35:45.878888 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.873009    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 23 11:35:45.878888 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.873013    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 23 11:35:45.878888 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.873017    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 23 11:35:45.878888 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.873022    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 23 11:35:45.878888 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.873025    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 23 11:35:45.878888 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.873030    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 23 11:35:45.878888 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.873034    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 23 11:35:45.878888 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.873038    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 23 11:35:45.878888 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.873042    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 23 11:35:45.878888 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.873046    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 23 11:35:45.878888 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.873050    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 23 11:35:45.879435 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.873054    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 23 11:35:45.879435 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.873062    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 23 11:35:45.879435 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.873067    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 23 11:35:45.879435 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.873071    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 23 11:35:45.879435 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.873075    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 23 11:35:45.879435 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.873079    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 23 11:35:45.879435 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.873083    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 23 11:35:45.879435 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.873087    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 23 11:35:45.879435 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.873092    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 23 11:35:45.879435 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.873097    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 23 11:35:45.879435 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.873101    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 23 11:35:45.879435 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.873105    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 23 11:35:45.879435 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.873109    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 23 11:35:45.879435 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.873114    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 23 11:35:45.879435 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.873122    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 23 11:35:45.879435 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.873126    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 23 11:35:45.879435 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.873131    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 23 11:35:45.879998 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.873139    2563 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 23 11:35:45.879998 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.874146    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 23 11:35:45.879998 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.874164    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 23 11:35:45.879998 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.874171    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 23 11:35:45.879998 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.874175    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 23 11:35:45.879998 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.874186    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 23 11:35:45.879998 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.874193    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 23 11:35:45.879998 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.874197    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 23 11:35:45.879998 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.874202    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 23 11:35:45.879998 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.874206    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 23 11:35:45.879998 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.874211    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 23 11:35:45.879998 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.874216    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 23 11:35:45.879998 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.874221    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 23 11:35:45.879998 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.874225    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 23 11:35:45.879998 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.874230    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 23 11:35:45.880372 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.874234    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 23 11:35:45.880372 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.874239    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 23 11:35:45.880372 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.874243    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 23 11:35:45.880372 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.874247    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 23 11:35:45.880372 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.874251    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 23 11:35:45.880372 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.874255    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 23 11:35:45.880372 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.874259    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 23 11:35:45.880372 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.874263    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 23 11:35:45.880372 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.874268    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 23 11:35:45.880372 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.874272    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 23 11:35:45.880372 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.874276    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 23 11:35:45.880372 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.874280    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 23 11:35:45.880372 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.874285    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 23 11:35:45.880372 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.874288    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 23 11:35:45.880372 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.874293    2563 feature_gate.go:328] unrecognized feature gate: Example
Jul 23 11:35:45.880372 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.874297    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 23 11:35:45.880372 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.874301    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 23 11:35:45.880372 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.874305    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 23 11:35:45.880372 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.874309    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 23 11:35:45.880372 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.874313    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 23 11:35:45.880902 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.874317    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 23 11:35:45.880902 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.874321    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 23 11:35:45.880902 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.874325    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 23 11:35:45.880902 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.874329    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 23 11:35:45.880902 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.874333    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 23 11:35:45.880902 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.874338    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 23 11:35:45.880902 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.874342    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 23 11:35:45.880902 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.874347    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 23 11:35:45.880902 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.874351    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 23 11:35:45.880902 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.874355    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 23 11:35:45.880902 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.874359    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 23 11:35:45.880902 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.874363    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 23 11:35:45.880902 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.874367    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 23 11:35:45.880902 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.874374    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 23 11:35:45.880902 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.874379    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 23 11:35:45.880902 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.874384    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 23 11:35:45.880902 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.874389    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 23 11:35:45.880902 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.874393    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 23 11:35:45.880902 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.874397    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 23 11:35:45.881609 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.874401    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 23 11:35:45.881609 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.874405    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 23 11:35:45.881609 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.874409    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 23 11:35:45.881609 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.874414    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 23 11:35:45.881609 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.874418    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 23 11:35:45.881609 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.874422    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 23 11:35:45.881609 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.874427    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 23 11:35:45.881609 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.874431    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 23 11:35:45.881609 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.874435    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 23 11:35:45.881609 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.874439    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 23 11:35:45.881609 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.874443    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 23 11:35:45.881609 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.874446    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 23 11:35:45.881609 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.874451    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 23 11:35:45.881609 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.874455    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 23 11:35:45.881609 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.874459    2563 feature_gate.go:328] unrecognized feature gate: Example2
Jul 23 11:35:45.881609 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.874463    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 23 11:35:45.881609 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.874467    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 23 11:35:45.881609 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.874471    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 23 11:35:45.881609 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.874476    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 23 11:35:45.881609 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.874480    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 23 11:35:45.882456 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.874484    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 23 11:35:45.882456 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.874488    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 23 11:35:45.882456 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.874492    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 23 11:35:45.882456 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.874496    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 23 11:35:45.882456 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.874500    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 23 11:35:45.882456 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.874504    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 23 11:35:45.882456 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.874507    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 23 11:35:45.882456 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.874511    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 23 11:35:45.882456 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.874516    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 23 11:35:45.882456 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.874521    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 23 11:35:45.882456 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.874527    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 23 11:35:45.882456 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.874533    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 23 11:35:45.882456 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.874538    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 23 11:35:45.882456 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.874542    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 23 11:35:45.882456 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.874546    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 23 11:35:45.882456 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.874550    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 23 11:35:45.882456 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.874554    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 23 11:35:45.882456 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.874558    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 23 11:35:45.882456 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.874562    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 23 11:35:45.882456 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.874566    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 23 11:35:45.883322 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.874570    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 23 11:35:45.883322 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.874575    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 23 11:35:45.883322 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.874579    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 23 11:35:45.883322 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.874583    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 23 11:35:45.883322 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.874589    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 23 11:35:45.883322 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:45.874593    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 23 11:35:45.883322 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.874602    2563 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 23 11:35:45.883322 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.874877    2563 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 23 11:35:45.883322 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.877590    2563 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 23 11:35:45.883322 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.879027    2563 server.go:1019] "Starting client certificate rotation"
Jul 23 11:35:45.883322 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.879124    2563 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 23 11:35:45.883322 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.879159    2563 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 23 11:35:45.883322 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.881709    2563 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 23 11:35:45.883753 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.883499    2563 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 23 11:35:45.894611 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.894593    2563 log.go:25] "Validated CRI v1 runtime API"
Jul 23 11:35:45.896722 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.896708    2563 log.go:25] "Validated CRI v1 image API"
Jul 23 11:35:45.897516 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.897499    2563 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 23 11:35:45.900221 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.900203    2563 fs.go:135] Filesystem UUIDs: map[018e07f8-b6f9-4d47-a29e-2d705612285c:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 a209a5e5-8793-47f0-ae73-207c5f6b0cf1:/dev/nvme0n1p4]
Jul 23 11:35:45.900270 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.900221    2563 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 23 11:35:45.904339 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.904238    2563 manager.go:216] Machine: {Timestamp:2026-07-23 11:35:45.903781704 +0000 UTC m=+0.296719483 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3112561 MemoryCapacity:32812179456 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec230aa69e7bad25df55f06d47a1f488 SystemUUID:ec230aa6-9e7b-ad25-df55-f06d47a1f488 BootID:da15e501-19da-4b05-982e-c59fa22dd730 Filesystems:[{Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406089728 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406089728 Type:vfs Inodes:4005393 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562439168 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:fc:fa:80:6a:fd Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:fc:fa:80:6a:fd Speed:0 Mtu:9001} {Name:ovs-system MacAddress:3a:08:74:aa:fa:c8 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812179456 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 23 11:35:45.904339 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.904337    2563 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 23 11:35:45.904472 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.904444    2563 manager.go:232] Version: {KernelVersion:5.14.0-570.128.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260715-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 23 11:35:45.907544 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.907517    2563 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 23 11:35:45.907706 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.907544    2563 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-141-229.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 23 11:35:45.907782 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.907768    2563 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 23 11:35:45.908791 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.908779    2563 topology_manager.go:138] "Creating topology manager with none policy"
Jul 23 11:35:45.908832 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.908794    2563 container_manager_linux.go:309] "Creating device plugin manager"
Jul 23 11:35:45.908832 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.908801    2563 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 23 11:35:45.909796 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.909784    2563 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 23 11:35:45.911097 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.911087    2563 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 23 11:35:45.914164 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.914153    2563 state_mem.go:36] "Initialized new in-memory state store"
Jul 23 11:35:45.914295 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.914286    2563 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 23 11:35:45.914327 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.914315    2563 kubelet.go:486] "Attempting to sync node with API server"
Jul 23 11:35:45.914327 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.914324    2563 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 23 11:35:45.914382 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.914338    2563 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 23 11:35:45.914382 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.914347    2563 kubelet.go:398] "Adding apiserver pod source"
Jul 23 11:35:45.914382 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.914355    2563 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 23 11:35:45.914811 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.914782    2563 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 23 11:35:45.914898 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.914819    2563 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 23 11:35:45.918472 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.918450    2563 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 23 11:35:45.920363 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.920347    2563 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 23 11:35:45.920418 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.920379    2563 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 23 11:35:45.921413 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.921402    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 23 11:35:45.921452 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.921419    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 23 11:35:45.921452 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.921429    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 23 11:35:45.921452 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.921437    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 23 11:35:45.921452 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.921450    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 23 11:35:45.921555 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.921456    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 23 11:35:45.921555 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.921465    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 23 11:35:45.921555 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.921470    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 23 11:35:45.921555 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.921477    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 23 11:35:45.921555 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.921482    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 23 11:35:45.921555 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.921490    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 23 11:35:45.921555 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.921496    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 23 11:35:45.922165 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.922155    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 23 11:35:45.922165 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.922165    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 23 11:35:45.925233 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.925211    2563 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-141-229.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 23 11:35:45.925589 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.925576    2563 server.go:1268] "Started kubelet"
Jul 23 11:35:45.925718 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.925679    2563 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 23 11:35:45.925800 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.925775    2563 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 23 11:35:45.925861 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.925820    2563 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 23 11:35:45.925914 ip-10-0-141-229 kubenswrapper[2563]: E0723 11:35:45.925890    2563 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-141-229.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 23 11:35:45.926025 ip-10-0-141-229 kubenswrapper[2563]: E0723 11:35:45.925909    2563 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 23 11:35:45.926446 ip-10-0-141-229 systemd[1]: Started Kubernetes Kubelet.
Jul 23 11:35:45.927420 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.927366    2563 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 23 11:35:45.927705 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.927676    2563 server.go:310] "Adding debug handlers to kubelet server"
Jul 23 11:35:45.930812 ip-10-0-141-229 kubenswrapper[2563]: E0723 11:35:45.929725    2563 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-229.ec2.internal.18c4e818e2af3479  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-229.ec2.internal,UID:ip-10-0-141-229.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-141-229.ec2.internal,},FirstTimestamp:2026-07-23 11:35:45.925551225 +0000 UTC m=+0.318489004,LastTimestamp:2026-07-23 11:35:45.925551225 +0000 UTC m=+0.318489004,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-229.ec2.internal,}"
Jul 23 11:35:45.931459 ip-10-0-141-229 kubenswrapper[2563]: E0723 11:35:45.931441    2563 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 23 11:35:45.932987 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.932971    2563 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-sf5dx"
Jul 23 11:35:45.934669 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.934656    2563 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 23 11:35:45.934747 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.934657    2563 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 23 11:35:45.934803 ip-10-0-141-229 kubenswrapper[2563]: E0723 11:35:45.934777    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-229.ec2.internal\" not found"
Jul 23 11:35:45.935348 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.935334    2563 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 23 11:35:45.935348 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.935351    2563 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 23 11:35:45.935482 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.935333    2563 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 23 11:35:45.935716 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.935489    2563 reconstruct.go:97] "Volume reconstruction finished"
Jul 23 11:35:45.935716 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.935496    2563 reconciler.go:29] "Reconciler: start to sync state"
Jul 23 11:35:45.935716 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.935583    2563 factory.go:55] Registering systemd factory
Jul 23 11:35:45.935716 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.935597    2563 factory.go:223] Registration of the systemd container factory successfully
Jul 23 11:35:45.935886 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.935872    2563 factory.go:153] Registering CRI-O factory
Jul 23 11:35:45.935886 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.935885    2563 factory.go:223] Registration of the crio container factory successfully
Jul 23 11:35:45.935969 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.935936    2563 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 23 11:35:45.935969 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.935960    2563 factory.go:103] Registering Raw factory
Jul 23 11:35:45.936065 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.935978    2563 manager.go:1196] Started watching for new ooms in manager
Jul 23 11:35:45.936384 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.936370    2563 manager.go:349] Starting recovery of all containers
Jul 23 11:35:45.936939 ip-10-0-141-229 kubenswrapper[2563]: E0723 11:35:45.936910    2563 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 23 11:35:45.937046 ip-10-0-141-229 kubenswrapper[2563]: E0723 11:35:45.936995    2563 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-141-229.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 23 11:35:45.941561 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.941537    2563 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-sf5dx"
Jul 23 11:35:45.947651 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.947629    2563 manager.go:354] Recovery completed
Jul 23 11:35:45.953614 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.953601    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 23 11:35:45.955574 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.955558    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-229.ec2.internal" event="NodeHasSufficientMemory"
Jul 23 11:35:45.955626 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.955589    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-229.ec2.internal" event="NodeHasNoDiskPressure"
Jul 23 11:35:45.955626 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.955602    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-229.ec2.internal" event="NodeHasSufficientPID"
Jul 23 11:35:45.956196 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.956183    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 23 11:35:45.956262 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.956188    2563 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 23 11:35:45.956262 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.956225    2563 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 23 11:35:45.956262 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.956241    2563 state_mem.go:36] "Initialized new in-memory state store"
Jul 23 11:35:45.956919 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.956899    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-229.ec2.internal" event="NodeHasSufficientMemory"
Jul 23 11:35:45.956996 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.956931    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-229.ec2.internal" event="NodeHasNoDiskPressure"
Jul 23 11:35:45.956996 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.956943    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-229.ec2.internal" event="NodeHasSufficientPID"
Jul 23 11:35:45.957362 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.957344    2563 policy_none.go:49] "None policy: Start"
Jul 23 11:35:45.957440 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.957364    2563 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 23 11:35:45.957440 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.957378    2563 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 23 11:35:45.958298 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.958284    2563 policy_none.go:47] "Start"
Jul 23 11:35:45.991106 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.990930    2563 manager.go:337] "Starting Device Plugin manager"
Jul 23 11:35:45.991229 ip-10-0-141-229 kubenswrapper[2563]: E0723 11:35:45.991135    2563 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 23 11:35:45.991229 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.991150    2563 server.go:87] "Starting device plugin registration server"
Jul 23 11:35:45.991392 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.991379    2563 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 23 11:35:45.991441 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.991395    2563 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 23 11:35:45.991586 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.991568    2563 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 23 11:35:45.991659 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.991636    2563 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 23 11:35:45.991659 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:45.991645    2563 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 23 11:35:45.993066 ip-10-0-141-229 kubenswrapper[2563]: E0723 11:35:45.993039    2563 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 23 11:35:45.993167 ip-10-0-141-229 kubenswrapper[2563]: E0723 11:35:45.993117    2563 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-141-229.ec2.internal\" not found"
Jul 23 11:35:46.070042 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:46.070002    2563 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 23 11:35:46.071377 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:46.071357    2563 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 23 11:35:46.071466 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:46.071399    2563 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 23 11:35:46.071466 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:46.071431    2563 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 23 11:35:46.071548 ip-10-0-141-229 kubenswrapper[2563]: E0723 11:35:46.071467    2563 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 23 11:35:46.073931 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:46.073907    2563 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 23 11:35:46.091905 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:46.091888    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 23 11:35:46.092869 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:46.092852    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-229.ec2.internal" event="NodeHasSufficientMemory"
Jul 23 11:35:46.092921 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:46.092885    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-229.ec2.internal" event="NodeHasNoDiskPressure"
Jul 23 11:35:46.092921 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:46.092900    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-229.ec2.internal" event="NodeHasSufficientPID"
Jul 23 11:35:46.092980 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:46.092924    2563 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-141-229.ec2.internal"
Jul 23 11:35:46.099612 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:46.099596    2563 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-141-229.ec2.internal"
Jul 23 11:35:46.099667 ip-10-0-141-229 kubenswrapper[2563]: E0723 11:35:46.099619    2563 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-141-229.ec2.internal\": node \"ip-10-0-141-229.ec2.internal\" not found"
Jul 23 11:35:46.114361 ip-10-0-141-229 kubenswrapper[2563]: E0723 11:35:46.114337    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-229.ec2.internal\" not found"
Jul 23 11:35:46.171937 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:46.171899    2563 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-229.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-141-229.ec2.internal"]
Jul 23 11:35:46.172079 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:46.171974    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 23 11:35:46.172983 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:46.172965    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-229.ec2.internal" event="NodeHasSufficientMemory"
Jul 23 11:35:46.173065 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:46.172998    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-229.ec2.internal" event="NodeHasNoDiskPressure"
Jul 23 11:35:46.173065 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:46.173007    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-229.ec2.internal" event="NodeHasSufficientPID"
Jul 23 11:35:46.174362 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:46.174350    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 23 11:35:46.174472 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:46.174448    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-229.ec2.internal"
Jul 23 11:35:46.174531 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:46.174479    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 23 11:35:46.175146 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:46.175119    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-229.ec2.internal" event="NodeHasSufficientMemory"
Jul 23 11:35:46.175146 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:46.175146    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-229.ec2.internal" event="NodeHasNoDiskPressure"
Jul 23 11:35:46.175274 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:46.175156    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-229.ec2.internal" event="NodeHasSufficientPID"
Jul 23 11:35:46.175274 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:46.175156    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-229.ec2.internal" event="NodeHasSufficientMemory"
Jul 23 11:35:46.175274 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:46.175177    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-229.ec2.internal" event="NodeHasNoDiskPressure"
Jul 23 11:35:46.175274 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:46.175200    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-229.ec2.internal" event="NodeHasSufficientPID"
Jul 23 11:35:46.176407 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:46.176392    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-229.ec2.internal"
Jul 23 11:35:46.176464 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:46.176418    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 23 11:35:46.177102 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:46.177082    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-229.ec2.internal" event="NodeHasSufficientMemory"
Jul 23 11:35:46.177215 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:46.177114    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-229.ec2.internal" event="NodeHasNoDiskPressure"
Jul 23 11:35:46.177215 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:46.177128    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-229.ec2.internal" event="NodeHasSufficientPID"
Jul 23 11:35:46.202273 ip-10-0-141-229 kubenswrapper[2563]: E0723 11:35:46.202250    2563 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-229.ec2.internal\" not found" node="ip-10-0-141-229.ec2.internal"
Jul 23 11:35:46.206648 ip-10-0-141-229 kubenswrapper[2563]: E0723 11:35:46.206634    2563 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-229.ec2.internal\" not found" node="ip-10-0-141-229.ec2.internal"
Jul 23 11:35:46.214693 ip-10-0-141-229 kubenswrapper[2563]: E0723 11:35:46.214667    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-229.ec2.internal\" not found"
Jul 23 11:35:46.237410 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:46.237379    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/192fe6abc0214ce0c06c38f944481f45-config\") pod \"kube-apiserver-proxy-ip-10-0-141-229.ec2.internal\" (UID: \"192fe6abc0214ce0c06c38f944481f45\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-229.ec2.internal"
Jul 23 11:35:46.237524 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:46.237415    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/92a19fc888e8076d0b3fbd4b9dc96a0a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-229.ec2.internal\" (UID: \"92a19fc888e8076d0b3fbd4b9dc96a0a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-229.ec2.internal"
Jul 23 11:35:46.237524 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:46.237442    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/92a19fc888e8076d0b3fbd4b9dc96a0a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-229.ec2.internal\" (UID: \"92a19fc888e8076d0b3fbd4b9dc96a0a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-229.ec2.internal"
Jul 23 11:35:46.315160 ip-10-0-141-229 kubenswrapper[2563]: E0723 11:35:46.315127    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-229.ec2.internal\" not found"
Jul 23 11:35:46.338506 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:46.338477    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/192fe6abc0214ce0c06c38f944481f45-config\") pod \"kube-apiserver-proxy-ip-10-0-141-229.ec2.internal\" (UID: \"192fe6abc0214ce0c06c38f944481f45\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-229.ec2.internal"
Jul 23 11:35:46.338571 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:46.338511    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/92a19fc888e8076d0b3fbd4b9dc96a0a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-229.ec2.internal\" (UID: \"92a19fc888e8076d0b3fbd4b9dc96a0a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-229.ec2.internal"
Jul 23 11:35:46.338571 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:46.338526    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/92a19fc888e8076d0b3fbd4b9dc96a0a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-229.ec2.internal\" (UID: \"92a19fc888e8076d0b3fbd4b9dc96a0a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-229.ec2.internal"
Jul 23 11:35:46.338571 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:46.338561    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/92a19fc888e8076d0b3fbd4b9dc96a0a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-229.ec2.internal\" (UID: \"92a19fc888e8076d0b3fbd4b9dc96a0a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-229.ec2.internal"
Jul 23 11:35:46.338678 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:46.338575    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/192fe6abc0214ce0c06c38f944481f45-config\") pod \"kube-apiserver-proxy-ip-10-0-141-229.ec2.internal\" (UID: \"192fe6abc0214ce0c06c38f944481f45\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-229.ec2.internal"
Jul 23 11:35:46.338678 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:46.338592    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/92a19fc888e8076d0b3fbd4b9dc96a0a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-229.ec2.internal\" (UID: \"92a19fc888e8076d0b3fbd4b9dc96a0a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-229.ec2.internal"
Jul 23 11:35:46.415903 ip-10-0-141-229 kubenswrapper[2563]: E0723 11:35:46.415823    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-229.ec2.internal\" not found"
Jul 23 11:35:46.506434 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:46.506399    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-229.ec2.internal"
Jul 23 11:35:46.509325 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:46.509307    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-229.ec2.internal"
Jul 23 11:35:46.516340 ip-10-0-141-229 kubenswrapper[2563]: E0723 11:35:46.516317    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-229.ec2.internal\" not found"
Jul 23 11:35:46.616873 ip-10-0-141-229 kubenswrapper[2563]: E0723 11:35:46.616833    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-229.ec2.internal\" not found"
Jul 23 11:35:46.717435 ip-10-0-141-229 kubenswrapper[2563]: E0723 11:35:46.717356    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-229.ec2.internal\" not found"
Jul 23 11:35:46.783627 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:46.783594    2563 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 23 11:35:46.835747 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:46.835723    2563 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-229.ec2.internal"
Jul 23 11:35:46.846023 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:46.846003    2563 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 23 11:35:46.847533 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:46.847521    2563 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-229.ec2.internal"
Jul 23 11:35:46.857616 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:46.857598    2563 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 23 11:35:46.878750 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:46.878730    2563 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 23 11:35:46.878877 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:46.878854    2563 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 23 11:35:46.878916 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:46.878873    2563 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 23 11:35:46.915088 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:46.915060    2563 apiserver.go:52] "Watching apiserver"
Jul 23 11:35:46.930572 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:46.930549    2563 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 23 11:35:46.933176 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:46.933151    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-target-l6jwv","openshift-network-operator/iptables-alerter-cb7gq","openshift-ovn-kubernetes/ovnkube-node-nhm9p","kube-system/global-pull-secret-syncer-cvbbt","kube-system/konnectivity-agent-2rs9k","kube-system/kube-apiserver-proxy-ip-10-0-141-229.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ljhws","openshift-cluster-node-tuning-operator/tuned-hxr6k","openshift-dns/node-resolver-dmnft","openshift-image-registry/node-ca-dg4zg","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-229.ec2.internal","openshift-multus/multus-additional-cni-plugins-prdbw","openshift-multus/multus-rknqz","openshift-multus/network-metrics-daemon-pbmtq"]
Jul 23 11:35:46.934473 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:46.934457    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l6jwv"
Jul 23 11:35:46.934533 ip-10-0-141-229 kubenswrapper[2563]: E0723 11:35:46.934520    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l6jwv" podUID="b30ea3bc-3059-4d68-934c-32cde3d06fbb"
Jul 23 11:35:46.935468 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:46.935444    2563 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 23 11:35:46.935578 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:46.935451    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-cb7gq"
Jul 23 11:35:46.938196 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:46.938142    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 23 11:35:46.938308 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:46.938281    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 23 11:35:46.938308 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:46.938295    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 23 11:35:46.938428 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:46.938298    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-9jgrc\""
Jul 23 11:35:46.939485 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:46.939465    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-nhm9p"
Jul 23 11:35:46.941003 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:46.940985    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cvbbt"
Jul 23 11:35:46.941127 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:46.941026    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2l6nr\" (UniqueName: \"kubernetes.io/projected/0bcf7e84-67a4-4cf6-bb1e-016bb38f4ccb-kube-api-access-2l6nr\") pod \"iptables-alerter-cb7gq\" (UID: \"0bcf7e84-67a4-4cf6-bb1e-016bb38f4ccb\") " pod="openshift-network-operator/iptables-alerter-cb7gq"
Jul 23 11:35:46.941127 ip-10-0-141-229 kubenswrapper[2563]: E0723 11:35:46.941051    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cvbbt" podUID="7a0946cf-0f04-49d5-ab74-6f64a48f8e5d"
Jul 23 11:35:46.941127 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:46.941075    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/93649ac9-9062-46c0-adaa-1048d471211a-host-run-netns\") pod \"ovnkube-node-nhm9p\" (UID: \"93649ac9-9062-46c0-adaa-1048d471211a\") " pod="openshift-ovn-kubernetes/ovnkube-node-nhm9p"
Jul 23 11:35:46.941127 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:46.941122    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/93649ac9-9062-46c0-adaa-1048d471211a-env-overrides\") pod \"ovnkube-node-nhm9p\" (UID: \"93649ac9-9062-46c0-adaa-1048d471211a\") " pod="openshift-ovn-kubernetes/ovnkube-node-nhm9p"
Jul 23 11:35:46.941343 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:46.941150    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/93649ac9-9062-46c0-adaa-1048d471211a-ovn-node-metrics-cert\") pod \"ovnkube-node-nhm9p\" (UID: \"93649ac9-9062-46c0-adaa-1048d471211a\") " pod="openshift-ovn-kubernetes/ovnkube-node-nhm9p"
Jul 23 11:35:46.941343 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:46.941167    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/93649ac9-9062-46c0-adaa-1048d471211a-ovnkube-script-lib\") pod \"ovnkube-node-nhm9p\" (UID: \"93649ac9-9062-46c0-adaa-1048d471211a\") " pod="openshift-ovn-kubernetes/ovnkube-node-nhm9p"
Jul 23 11:35:46.941343 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:46.941181    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/93649ac9-9062-46c0-adaa-1048d471211a-systemd-units\") pod \"ovnkube-node-nhm9p\" (UID: \"93649ac9-9062-46c0-adaa-1048d471211a\") " pod="openshift-ovn-kubernetes/ovnkube-node-nhm9p"
Jul 23 11:35:46.941343 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:46.941199    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/93649ac9-9062-46c0-adaa-1048d471211a-host-slash\") pod \"ovnkube-node-nhm9p\" (UID: \"93649ac9-9062-46c0-adaa-1048d471211a\") " pod="openshift-ovn-kubernetes/ovnkube-node-nhm9p"
Jul 23 11:35:46.941343 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:46.941221    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/93649ac9-9062-46c0-adaa-1048d471211a-run-ovn\") pod \"ovnkube-node-nhm9p\" (UID: \"93649ac9-9062-46c0-adaa-1048d471211a\") " pod="openshift-ovn-kubernetes/ovnkube-node-nhm9p"
Jul 23 11:35:46.941343 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:46.941242    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/93649ac9-9062-46c0-adaa-1048d471211a-node-log\") pod \"ovnkube-node-nhm9p\" (UID: \"93649ac9-9062-46c0-adaa-1048d471211a\") " pod="openshift-ovn-kubernetes/ovnkube-node-nhm9p"
Jul 23 11:35:46.941343 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:46.941264    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/93649ac9-9062-46c0-adaa-1048d471211a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-nhm9p\" (UID: \"93649ac9-9062-46c0-adaa-1048d471211a\") " pod="openshift-ovn-kubernetes/ovnkube-node-nhm9p"
Jul 23 11:35:46.941343 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:46.941285    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-528jd\" (UniqueName: \"kubernetes.io/projected/b30ea3bc-3059-4d68-934c-32cde3d06fbb-kube-api-access-528jd\") pod \"network-check-target-l6jwv\" (UID: \"b30ea3bc-3059-4d68-934c-32cde3d06fbb\") " pod="openshift-network-diagnostics/network-check-target-l6jwv"
Jul 23 11:35:46.941343 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:46.941332    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/93649ac9-9062-46c0-adaa-1048d471211a-var-lib-openvswitch\") pod \"ovnkube-node-nhm9p\" (UID: \"93649ac9-9062-46c0-adaa-1048d471211a\") " pod="openshift-ovn-kubernetes/ovnkube-node-nhm9p"
Jul 23 11:35:46.941833 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:46.941368    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/93649ac9-9062-46c0-adaa-1048d471211a-run-openvswitch\") pod \"ovnkube-node-nhm9p\" (UID: \"93649ac9-9062-46c0-adaa-1048d471211a\") " pod="openshift-ovn-kubernetes/ovnkube-node-nhm9p"
Jul 23 11:35:46.941833 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:46.941395    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0bcf7e84-67a4-4cf6-bb1e-016bb38f4ccb-iptables-alerter-script\") pod \"iptables-alerter-cb7gq\" (UID: \"0bcf7e84-67a4-4cf6-bb1e-016bb38f4ccb\") " pod="openshift-network-operator/iptables-alerter-cb7gq"
Jul 23 11:35:46.941833 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:46.941418    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0bcf7e84-67a4-4cf6-bb1e-016bb38f4ccb-host-slash\") pod \"iptables-alerter-cb7gq\" (UID: \"0bcf7e84-67a4-4cf6-bb1e-016bb38f4ccb\") " pod="openshift-network-operator/iptables-alerter-cb7gq"
Jul 23 11:35:46.941833 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:46.941439    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/93649ac9-9062-46c0-adaa-1048d471211a-run-systemd\") pod \"ovnkube-node-nhm9p\" (UID: \"93649ac9-9062-46c0-adaa-1048d471211a\") " pod="openshift-ovn-kubernetes/ovnkube-node-nhm9p"
Jul 23 11:35:46.941833 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:46.941459    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/93649ac9-9062-46c0-adaa-1048d471211a-log-socket\") pod \"ovnkube-node-nhm9p\" (UID: \"93649ac9-9062-46c0-adaa-1048d471211a\") " pod="openshift-ovn-kubernetes/ovnkube-node-nhm9p"
Jul 23 11:35:46.941833 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:46.941493    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/93649ac9-9062-46c0-adaa-1048d471211a-host-cni-bin\") pod \"ovnkube-node-nhm9p\" (UID: \"93649ac9-9062-46c0-adaa-1048d471211a\") " pod="openshift-ovn-kubernetes/ovnkube-node-nhm9p"
Jul 23 11:35:46.941833 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:46.941546    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/93649ac9-9062-46c0-adaa-1048d471211a-host-kubelet\") pod \"ovnkube-node-nhm9p\" (UID: \"93649ac9-9062-46c0-adaa-1048d471211a\") " pod="openshift-ovn-kubernetes/ovnkube-node-nhm9p"
Jul 23 11:35:46.941833 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:46.941571    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/93649ac9-9062-46c0-adaa-1048d471211a-host-run-ovn-kubernetes\") pod \"ovnkube-node-nhm9p\" (UID: \"93649ac9-9062-46c0-adaa-1048d471211a\") " pod="openshift-ovn-kubernetes/ovnkube-node-nhm9p"
Jul 23 11:35:46.941833 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:46.941602    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/93649ac9-9062-46c0-adaa-1048d471211a-host-cni-netd\") pod \"ovnkube-node-nhm9p\" (UID: \"93649ac9-9062-46c0-adaa-1048d471211a\") " pod="openshift-ovn-kubernetes/ovnkube-node-nhm9p"
Jul 23 11:35:46.941833 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:46.941623    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 23 11:35:46.941833 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:46.941626    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/93649ac9-9062-46c0-adaa-1048d471211a-ovnkube-config\") pod \"ovnkube-node-nhm9p\" (UID: \"93649ac9-9062-46c0-adaa-1048d471211a\") " pod="openshift-ovn-kubernetes/ovnkube-node-nhm9p"
Jul 23 11:35:46.941833 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:46.941716    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxwcn\" (UniqueName: \"kubernetes.io/projected/93649ac9-9062-46c0-adaa-1048d471211a-kube-api-access-hxwcn\") pod \"ovnkube-node-nhm9p\" (UID: \"93649ac9-9062-46c0-adaa-1048d471211a\") " pod="openshift-ovn-kubernetes/ovnkube-node-nhm9p"
Jul 23 11:35:46.941833 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:46.941741    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/93649ac9-9062-46c0-adaa-1048d471211a-etc-openvswitch\") pod \"ovnkube-node-nhm9p\" (UID: \"93649ac9-9062-46c0-adaa-1048d471211a\") " pod="openshift-ovn-kubernetes/ovnkube-node-nhm9p"
Jul 23 11:35:46.942292 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:46.942183    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-l49kp\""
Jul 23 11:35:46.942326 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:46.942319    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 23 11:35:46.942511 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:46.942486    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 23 11:35:46.942610 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:46.942518    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-2rs9k"
Jul 23 11:35:46.942796 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:46.942772    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 23 11:35:46.942903 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:46.942779    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 23 11:35:46.942982 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:46.942968    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 23 11:35:46.943729 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:46.943714    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ljhws"
Jul 23 11:35:46.944735 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:46.944719    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-qcvpg\""
Jul 23 11:35:46.945300 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:46.945285    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-hxr6k"
Jul 23 11:35:46.946257 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:46.946233    2563 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-22 11:30:46 +0000 UTC" deadline="2028-03-16 12:38:59.749219484 +0000 UTC"
Jul 23 11:35:46.946257 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:46.946255    2563 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14449h3m12.802967082s"
Jul 23 11:35:46.946438 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:46.946419    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 23 11:35:46.946504 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:46.946426    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 23 11:35:46.946504 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:46.946483    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 23 11:35:46.946504 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:46.946495    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-gzxvp\""
Jul 23 11:35:46.946726 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:46.946535    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 23 11:35:46.946726 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:46.946554    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 23 11:35:46.947031 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:46.947013    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-dmnft"
Jul 23 11:35:46.947279 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:46.947259    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 23 11:35:46.947381 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:46.947270    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 23 11:35:46.947381 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:46.947260    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-twz25\""
Jul 23 11:35:46.948772 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:46.948757    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-dg4zg"
Jul 23 11:35:46.949286 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:46.949268    2563 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 23 11:35:46.949457 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:46.949441    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 23 11:35:46.950103 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:46.950087    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 23 11:35:46.950193 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:46.950135    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-prdbw"
Jul 23 11:35:46.951287 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:46.951268    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-qbvb9\""
Jul 23 11:35:46.951363 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:46.951345    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 23 11:35:46.951725 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:46.951700    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-rknqz"
Jul 23 11:35:46.951852 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:46.951809    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 23 11:35:46.952097 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:46.952065    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 23 11:35:46.952183 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:46.952105    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-dsntn\""
Jul 23 11:35:46.953048 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:46.953030    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pbmtq"
Jul 23 11:35:46.953169 ip-10-0-141-229 kubenswrapper[2563]: E0723 11:35:46.953101    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pbmtq" podUID="cbdff1cb-47fd-4ed9-bec9-419ea8da3aaa"
Jul 23 11:35:46.953666 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:46.953650    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 23 11:35:46.953812 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:46.953794    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-nzlgs\""
Jul 23 11:35:46.953945 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:46.953927    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 23 11:35:46.954896 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:46.954365    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 23 11:35:46.954896 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:46.954562    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 23 11:35:46.954896 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:46.954832    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 23 11:35:46.955739 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:46.955722    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-xj877\""
Jul 23 11:35:46.955818 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:46.955756    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 23 11:35:46.978914 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:46.978896    2563 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-5smsz"
Jul 23 11:35:46.985343 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:46.985323    2563 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-5smsz"
Jul 23 11:35:47.036175 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:47.036151    2563 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 23 11:35:47.042103 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:47.042074    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e6553d8c-9a9e-40be-a703-77d9c5235122-var-lib-kubelet\") pod \"tuned-hxr6k\" (UID: \"e6553d8c-9a9e-40be-a703-77d9c5235122\") " pod="openshift-cluster-node-tuning-operator/tuned-hxr6k"
Jul 23 11:35:47.042218 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:47.042124    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/93649ac9-9062-46c0-adaa-1048d471211a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-nhm9p\" (UID: \"93649ac9-9062-46c0-adaa-1048d471211a\") " pod="openshift-ovn-kubernetes/ovnkube-node-nhm9p"
Jul 23 11:35:47.042218 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:47.042159    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/7a0946cf-0f04-49d5-ab74-6f64a48f8e5d-kubelet-config\") pod \"global-pull-secret-syncer-cvbbt\" (UID: \"7a0946cf-0f04-49d5-ab74-6f64a48f8e5d\") " pod="kube-system/global-pull-secret-syncer-cvbbt"
Jul 23 11:35:47.042218 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:47.042185    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/7a0946cf-0f04-49d5-ab74-6f64a48f8e5d-dbus\") pod \"global-pull-secret-syncer-cvbbt\" (UID: \"7a0946cf-0f04-49d5-ab74-6f64a48f8e5d\") " pod="kube-system/global-pull-secret-syncer-cvbbt"
Jul 23 11:35:47.042218 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:47.042207    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/93649ac9-9062-46c0-adaa-1048d471211a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-nhm9p\" (UID: \"93649ac9-9062-46c0-adaa-1048d471211a\") " pod="openshift-ovn-kubernetes/ovnkube-node-nhm9p"
Jul 23 11:35:47.042467 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:47.042210    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e6553d8c-9a9e-40be-a703-77d9c5235122-run\") pod \"tuned-hxr6k\" (UID: \"e6553d8c-9a9e-40be-a703-77d9c5235122\") " pod="openshift-cluster-node-tuning-operator/tuned-hxr6k"
Jul 23 11:35:47.042467 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:47.042277    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e6553d8c-9a9e-40be-a703-77d9c5235122-host\") pod \"tuned-hxr6k\" (UID: \"e6553d8c-9a9e-40be-a703-77d9c5235122\") " pod="openshift-cluster-node-tuning-operator/tuned-hxr6k"
Jul 23 11:35:47.042467 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:47.042300    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-blmdc\" (UniqueName: \"kubernetes.io/projected/e6553d8c-9a9e-40be-a703-77d9c5235122-kube-api-access-blmdc\") pod \"tuned-hxr6k\" (UID: \"e6553d8c-9a9e-40be-a703-77d9c5235122\") " pod="openshift-cluster-node-tuning-operator/tuned-hxr6k"
Jul 23 11:35:47.042467 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:47.042316    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0075cfcb-3377-4888-b93b-1fe1e8f4b6da-hostroot\") pod \"multus-rknqz\" (UID: \"0075cfcb-3377-4888-b93b-1fe1e8f4b6da\") " pod="openshift-multus/multus-rknqz"
Jul 23 11:35:47.042467 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:47.042340    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0bcf7e84-67a4-4cf6-bb1e-016bb38f4ccb-host-slash\") pod \"iptables-alerter-cb7gq\" (UID: \"0bcf7e84-67a4-4cf6-bb1e-016bb38f4ccb\") " pod="openshift-network-operator/iptables-alerter-cb7gq"
Jul 23 11:35:47.042467 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:47.042366    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/7a2f8e1f-040a-4e01-88a7-9909e2fbc8b3-os-release\") pod \"multus-additional-cni-plugins-prdbw\" (UID: \"7a2f8e1f-040a-4e01-88a7-9909e2fbc8b3\") " pod="openshift-multus/multus-additional-cni-plugins-prdbw"
Jul 23 11:35:47.042467 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:47.042383    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0075cfcb-3377-4888-b93b-1fe1e8f4b6da-multus-cni-dir\") pod \"multus-rknqz\" (UID: \"0075cfcb-3377-4888-b93b-1fe1e8f4b6da\") " pod="openshift-multus/multus-rknqz"
Jul 23 11:35:47.042467 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:47.042429    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0075cfcb-3377-4888-b93b-1fe1e8f4b6da-host-var-lib-cni-bin\") pod \"multus-rknqz\" (UID: \"0075cfcb-3377-4888-b93b-1fe1e8f4b6da\") " pod="openshift-multus/multus-rknqz"
Jul 23 11:35:47.042467 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:47.042434    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0bcf7e84-67a4-4cf6-bb1e-016bb38f4ccb-host-slash\") pod \"iptables-alerter-cb7gq\" (UID: \"0bcf7e84-67a4-4cf6-bb1e-016bb38f4ccb\") " pod="openshift-network-operator/iptables-alerter-cb7gq"
Jul 23 11:35:47.042467 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:47.042468    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxpkb\" (UniqueName: \"kubernetes.io/projected/0075cfcb-3377-4888-b93b-1fe1e8f4b6da-kube-api-access-qxpkb\") pod \"multus-rknqz\" (UID: \"0075cfcb-3377-4888-b93b-1fe1e8f4b6da\") " pod="openshift-multus/multus-rknqz"
Jul 23 11:35:47.043007 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:47.042501    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/93649ac9-9062-46c0-adaa-1048d471211a-host-kubelet\") pod \"ovnkube-node-nhm9p\" (UID: \"93649ac9-9062-46c0-adaa-1048d471211a\") " pod="openshift-ovn-kubernetes/ovnkube-node-nhm9p"
Jul 23 11:35:47.043007 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:47.042529    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/93649ac9-9062-46c0-adaa-1048d471211a-host-run-ovn-kubernetes\") pod \"ovnkube-node-nhm9p\" (UID: \"93649ac9-9062-46c0-adaa-1048d471211a\") " pod="openshift-ovn-kubernetes/ovnkube-node-nhm9p"
Jul 23 11:35:47.043007 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:47.042554    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e6553d8c-9a9e-40be-a703-77d9c5235122-sys\") pod \"tuned-hxr6k\" (UID: \"e6553d8c-9a9e-40be-a703-77d9c5235122\") " pod="openshift-cluster-node-tuning-operator/tuned-hxr6k"
Jul 23 11:35:47.043007 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:47.042562    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/93649ac9-9062-46c0-adaa-1048d471211a-host-kubelet\") pod \"ovnkube-node-nhm9p\" (UID: \"93649ac9-9062-46c0-adaa-1048d471211a\") " pod="openshift-ovn-kubernetes/ovnkube-node-nhm9p"
Jul 23 11:35:47.043007 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:47.042582    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e6553d8c-9a9e-40be-a703-77d9c5235122-lib-modules\") pod \"tuned-hxr6k\" (UID: \"e6553d8c-9a9e-40be-a703-77d9c5235122\") " pod="openshift-cluster-node-tuning-operator/tuned-hxr6k"
Jul 23 11:35:47.043007 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:47.042602    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/93649ac9-9062-46c0-adaa-1048d471211a-host-run-ovn-kubernetes\") pod \"ovnkube-node-nhm9p\" (UID: \"93649ac9-9062-46c0-adaa-1048d471211a\") " pod="openshift-ovn-kubernetes/ovnkube-node-nhm9p"
Jul 23 11:35:47.043007 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:47.042606    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e6553d8c-9a9e-40be-a703-77d9c5235122-tmp\") pod \"tuned-hxr6k\" (UID: \"e6553d8c-9a9e-40be-a703-77d9c5235122\") " pod="openshift-cluster-node-tuning-operator/tuned-hxr6k"
Jul 23 11:35:47.043007 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:47.042647    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2ba42487-6ccf-4e15-93df-65f0623c41ba-host\") pod \"node-ca-dg4zg\" (UID: \"2ba42487-6ccf-4e15-93df-65f0623c41ba\") " pod="openshift-image-registry/node-ca-dg4zg"
Jul 23 11:35:47.043007 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:47.042673    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0075cfcb-3377-4888-b93b-1fe1e8f4b6da-host-var-lib-kubelet\") pod \"multus-rknqz\" (UID: \"0075cfcb-3377-4888-b93b-1fe1e8f4b6da\") " pod="openshift-multus/multus-rknqz"
Jul 23 11:35:47.043007 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:47.042703    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2l6nr\" (UniqueName: \"kubernetes.io/projected/0bcf7e84-67a4-4cf6-bb1e-016bb38f4ccb-kube-api-access-2l6nr\") pod \"iptables-alerter-cb7gq\" (UID: \"0bcf7e84-67a4-4cf6-bb1e-016bb38f4ccb\") " pod="openshift-network-operator/iptables-alerter-cb7gq"
Jul 23 11:35:47.043007 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:47.042729    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e6553d8c-9a9e-40be-a703-77d9c5235122-etc-modprobe-d\") pod \"tuned-hxr6k\" (UID: \"e6553d8c-9a9e-40be-a703-77d9c5235122\") " pod="openshift-cluster-node-tuning-operator/tuned-hxr6k"
Jul 23 11:35:47.043007 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:47.042776    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/93649ac9-9062-46c0-adaa-1048d471211a-run-ovn\") pod \"ovnkube-node-nhm9p\" (UID: \"93649ac9-9062-46c0-adaa-1048d471211a\") " pod="openshift-ovn-kubernetes/ovnkube-node-nhm9p"
Jul 23 11:35:47.043007 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:47.042804    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s75l6\" (UniqueName: \"kubernetes.io/projected/6bf989cb-53d0-4895-8bd1-377fe1451b7b-kube-api-access-s75l6\") pod \"aws-ebs-csi-driver-node-ljhws\" (UID: \"6bf989cb-53d0-4895-8bd1-377fe1451b7b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ljhws"
Jul 23 11:35:47.043007 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:47.042829    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0075cfcb-3377-4888-b93b-1fe1e8f4b6da-host-run-netns\") pod \"multus-rknqz\" (UID: \"0075cfcb-3377-4888-b93b-1fe1e8f4b6da\") " pod="openshift-multus/multus-rknqz"
Jul 23 11:35:47.043007 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:47.042855    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/93649ac9-9062-46c0-adaa-1048d471211a-run-openvswitch\") pod \"ovnkube-node-nhm9p\" (UID: \"93649ac9-9062-46c0-adaa-1048d471211a\") " pod="openshift-ovn-kubernetes/ovnkube-node-nhm9p"
Jul 23 11:35:47.043007 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:47.042867    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/93649ac9-9062-46c0-adaa-1048d471211a-run-ovn\") pod \"ovnkube-node-nhm9p\" (UID: \"93649ac9-9062-46c0-adaa-1048d471211a\") " pod="openshift-ovn-kubernetes/ovnkube-node-nhm9p"
Jul 23 11:35:47.043007 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:47.042880    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/93649ac9-9062-46c0-adaa-1048d471211a-run-systemd\") pod \"ovnkube-node-nhm9p\" (UID: \"93649ac9-9062-46c0-adaa-1048d471211a\") " pod="openshift-ovn-kubernetes/ovnkube-node-nhm9p"
Jul 23 11:35:47.043985 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:47.042904    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/93649ac9-9062-46c0-adaa-1048d471211a-log-socket\") pod \"ovnkube-node-nhm9p\" (UID: \"93649ac9-9062-46c0-adaa-1048d471211a\") " pod="openshift-ovn-kubernetes/ovnkube-node-nhm9p"
Jul 23 11:35:47.043985 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:47.042931    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7a0946cf-0f04-49d5-ab74-6f64a48f8e5d-original-pull-secret\") pod \"global-pull-secret-syncer-cvbbt\" (UID: \"7a0946cf-0f04-49d5-ab74-6f64a48f8e5d\") " pod="kube-system/global-pull-secret-syncer-cvbbt"
Jul 23 11:35:47.043985 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:47.042957    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7a2f8e1f-040a-4e01-88a7-9909e2fbc8b3-cni-binary-copy\") pod \"multus-additional-cni-plugins-prdbw\" (UID: \"7a2f8e1f-040a-4e01-88a7-9909e2fbc8b3\") " pod="openshift-multus/multus-additional-cni-plugins-prdbw"
Jul 23 11:35:47.043985 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:47.042957    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/93649ac9-9062-46c0-adaa-1048d471211a-run-openvswitch\") pod \"ovnkube-node-nhm9p\" (UID: \"93649ac9-9062-46c0-adaa-1048d471211a\") " pod="openshift-ovn-kubernetes/ovnkube-node-nhm9p"
Jul 23 11:35:47.043985 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:47.042981    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0075cfcb-3377-4888-b93b-1fe1e8f4b6da-system-cni-dir\") pod \"multus-rknqz\" (UID: \"0075cfcb-3377-4888-b93b-1fe1e8f4b6da\") " pod="openshift-multus/multus-rknqz"
Jul 23 11:35:47.043985 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:47.043011    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/93649ac9-9062-46c0-adaa-1048d471211a-log-socket\") pod \"ovnkube-node-nhm9p\" (UID: \"93649ac9-9062-46c0-adaa-1048d471211a\") " pod="openshift-ovn-kubernetes/ovnkube-node-nhm9p"
Jul 23 11:35:47.043985 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:47.043011    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/93649ac9-9062-46c0-adaa-1048d471211a-ovnkube-config\") pod \"ovnkube-node-nhm9p\" (UID: \"93649ac9-9062-46c0-adaa-1048d471211a\") " pod="openshift-ovn-kubernetes/ovnkube-node-nhm9p"
Jul 23 11:35:47.043985 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:47.043056    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/6bf989cb-53d0-4895-8bd1-377fe1451b7b-socket-dir\") pod \"aws-ebs-csi-driver-node-ljhws\" (UID: \"6bf989cb-53d0-4895-8bd1-377fe1451b7b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ljhws"
Jul 23 11:35:47.043985 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:47.043165    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/93649ac9-9062-46c0-adaa-1048d471211a-run-systemd\") pod \"ovnkube-node-nhm9p\" (UID: \"93649ac9-9062-46c0-adaa-1048d471211a\") " pod="openshift-ovn-kubernetes/ovnkube-node-nhm9p"
Jul 23 11:35:47.043985 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:47.043178    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8kc6s\" (UniqueName: \"kubernetes.io/projected/5ef3b70f-a117-4dcd-9380-5c5d36e596c0-kube-api-access-8kc6s\") pod \"node-resolver-dmnft\" (UID: \"5ef3b70f-a117-4dcd-9380-5c5d36e596c0\") " pod="openshift-dns/node-resolver-dmnft"
Jul 23 11:35:47.043985 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:47.043223    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/73f77fae-fdff-4232-a26a-e5749f7df186-konnectivity-ca\") pod \"konnectivity-agent-2rs9k\" (UID: \"73f77fae-fdff-4232-a26a-e5749f7df186\") " pod="kube-system/konnectivity-agent-2rs9k"
Jul 23 11:35:47.043985 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:47.043249    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/7a2f8e1f-040a-4e01-88a7-9909e2fbc8b3-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-prdbw\" (UID: \"7a2f8e1f-040a-4e01-88a7-9909e2fbc8b3\") " pod="openshift-multus/multus-additional-cni-plugins-prdbw"
Jul 23 11:35:47.043985 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:47.043286    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/93649ac9-9062-46c0-adaa-1048d471211a-ovn-node-metrics-cert\") pod \"ovnkube-node-nhm9p\" (UID: \"93649ac9-9062-46c0-adaa-1048d471211a\") " pod="openshift-ovn-kubernetes/ovnkube-node-nhm9p"
Jul 23 11:35:47.043985 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:47.043336    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/93649ac9-9062-46c0-adaa-1048d471211a-ovnkube-script-lib\") pod \"ovnkube-node-nhm9p\" (UID: \"93649ac9-9062-46c0-adaa-1048d471211a\") " pod="openshift-ovn-kubernetes/ovnkube-node-nhm9p"
Jul 23 11:35:47.043985 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:47.043356    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e6553d8c-9a9e-40be-a703-77d9c5235122-etc-tuned\") pod \"tuned-hxr6k\" (UID: \"e6553d8c-9a9e-40be-a703-77d9c5235122\") " pod="openshift-cluster-node-tuning-operator/tuned-hxr6k"
Jul 23 11:35:47.043985 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:47.043371    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/5ef3b70f-a117-4dcd-9380-5c5d36e596c0-hosts-file\") pod \"node-resolver-dmnft\" (UID: \"5ef3b70f-a117-4dcd-9380-5c5d36e596c0\") " pod="openshift-dns/node-resolver-dmnft"
Jul 23 11:35:47.043985 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:47.043385    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/0075cfcb-3377-4888-b93b-1fe1e8f4b6da-os-release\") pod \"multus-rknqz\" (UID: \"0075cfcb-3377-4888-b93b-1fe1e8f4b6da\") " pod="openshift-multus/multus-rknqz"
Jul 23 11:35:47.044669 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:47.043401    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/93649ac9-9062-46c0-adaa-1048d471211a-systemd-units\") pod \"ovnkube-node-nhm9p\" (UID: \"93649ac9-9062-46c0-adaa-1048d471211a\") " pod="openshift-ovn-kubernetes/ovnkube-node-nhm9p"
Jul 23 11:35:47.044669 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:47.043428    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/93649ac9-9062-46c0-adaa-1048d471211a-node-log\") pod \"ovnkube-node-nhm9p\" (UID: \"93649ac9-9062-46c0-adaa-1048d471211a\") " pod="openshift-ovn-kubernetes/ovnkube-node-nhm9p"
Jul 23 11:35:47.044669 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:47.043530    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/93649ac9-9062-46c0-adaa-1048d471211a-node-log\") pod \"ovnkube-node-nhm9p\" (UID: \"93649ac9-9062-46c0-adaa-1048d471211a\") " pod="openshift-ovn-kubernetes/ovnkube-node-nhm9p"
Jul 23 11:35:47.044669 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:47.043535    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e6553d8c-9a9e-40be-a703-77d9c5235122-etc-sysctl-conf\") pod \"tuned-hxr6k\" (UID: \"e6553d8c-9a9e-40be-a703-77d9c5235122\") " pod="openshift-cluster-node-tuning-operator/tuned-hxr6k"
Jul 23 11:35:47.044669 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:47.043578    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/7a2f8e1f-040a-4e01-88a7-9909e2fbc8b3-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-prdbw\" (UID: \"7a2f8e1f-040a-4e01-88a7-9909e2fbc8b3\") " pod="openshift-multus/multus-additional-cni-plugins-prdbw"
Jul 23 11:35:47.044669 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:47.043633    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/93649ac9-9062-46c0-adaa-1048d471211a-systemd-units\") pod \"ovnkube-node-nhm9p\" (UID: \"93649ac9-9062-46c0-adaa-1048d471211a\") " pod="openshift-ovn-kubernetes/ovnkube-node-nhm9p"
Jul 23 11:35:47.044669 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:47.043637    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0075cfcb-3377-4888-b93b-1fe1e8f4b6da-cni-binary-copy\") pod \"multus-rknqz\" (UID: \"0075cfcb-3377-4888-b93b-1fe1e8f4b6da\") " pod="openshift-multus/multus-rknqz"
Jul 23 11:35:47.044669 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:47.043640    2563 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 23 11:35:47.044669 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:47.043679    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/93649ac9-9062-46c0-adaa-1048d471211a-var-lib-openvswitch\") pod \"ovnkube-node-nhm9p\" (UID: \"93649ac9-9062-46c0-adaa-1048d471211a\") " pod="openshift-ovn-kubernetes/ovnkube-node-nhm9p"
Jul 23 11:35:47.044669 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:47.043749    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/93649ac9-9062-46c0-adaa-1048d471211a-var-lib-openvswitch\") pod \"ovnkube-node-nhm9p\" (UID: \"93649ac9-9062-46c0-adaa-1048d471211a\") " pod="openshift-ovn-kubernetes/ovnkube-node-nhm9p"
Jul 23 11:35:47.044669 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:47.043781    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0bcf7e84-67a4-4cf6-bb1e-016bb38f4ccb-iptables-alerter-script\") pod \"iptables-alerter-cb7gq\" (UID: \"0bcf7e84-67a4-4cf6-bb1e-016bb38f4ccb\") " pod="openshift-network-operator/iptables-alerter-cb7gq"
Jul 23 11:35:47.044669 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:47.043759    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/93649ac9-9062-46c0-adaa-1048d471211a-ovnkube-config\") pod \"ovnkube-node-nhm9p\" (UID: \"93649ac9-9062-46c0-adaa-1048d471211a\") " pod="openshift-ovn-kubernetes/ovnkube-node-nhm9p"
Jul 23 11:35:47.044669 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:47.043825    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/93649ac9-9062-46c0-adaa-1048d471211a-host-cni-bin\") pod \"ovnkube-node-nhm9p\" (UID: \"93649ac9-9062-46c0-adaa-1048d471211a\") " pod="openshift-ovn-kubernetes/ovnkube-node-nhm9p"
Jul 23 11:35:47.044669 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:47.043854    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/93649ac9-9062-46c0-adaa-1048d471211a-host-cni-netd\") pod \"ovnkube-node-nhm9p\" (UID: \"93649ac9-9062-46c0-adaa-1048d471211a\") " pod="openshift-ovn-kubernetes/ovnkube-node-nhm9p"
Jul 23 11:35:47.044669 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:47.043881    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxwcn\" (UniqueName: \"kubernetes.io/projected/93649ac9-9062-46c0-adaa-1048d471211a-kube-api-access-hxwcn\") pod \"ovnkube-node-nhm9p\" (UID: \"93649ac9-9062-46c0-adaa-1048d471211a\") " pod="openshift-ovn-kubernetes/ovnkube-node-nhm9p"
Jul 23 11:35:47.044669 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:47.043933    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/6bf989cb-53d0-4895-8bd1-377fe1451b7b-registration-dir\") pod \"aws-ebs-csi-driver-node-ljhws\" (UID: \"6bf989cb-53d0-4895-8bd1-377fe1451b7b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ljhws"
Jul 23 11:35:47.044669 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:47.043969    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/6bf989cb-53d0-4895-8bd1-377fe1451b7b-etc-selinux\") pod \"aws-ebs-csi-driver-node-ljhws\" (UID: \"6bf989cb-53d0-4895-8bd1-377fe1451b7b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ljhws"
Jul 23 11:35:47.045423 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:47.043992    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/93649ac9-9062-46c0-adaa-1048d471211a-ovnkube-script-lib\") pod \"ovnkube-node-nhm9p\" (UID: \"93649ac9-9062-46c0-adaa-1048d471211a\") " pod="openshift-ovn-kubernetes/ovnkube-node-nhm9p"
Jul 23 11:35:47.045423 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:47.043994    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e6553d8c-9a9e-40be-a703-77d9c5235122-etc-sysconfig\") pod \"tuned-hxr6k\" (UID: \"e6553d8c-9a9e-40be-a703-77d9c5235122\") " pod="openshift-cluster-node-tuning-operator/tuned-hxr6k"
Jul 23 11:35:47.045423 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:47.044040    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/93649ac9-9062-46c0-adaa-1048d471211a-host-cni-bin\") pod \"ovnkube-node-nhm9p\" (UID: \"93649ac9-9062-46c0-adaa-1048d471211a\") " pod="openshift-ovn-kubernetes/ovnkube-node-nhm9p"
Jul 23 11:35:47.045423 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:47.044058    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/6bf989cb-53d0-4895-8bd1-377fe1451b7b-sys-fs\") pod \"aws-ebs-csi-driver-node-ljhws\" (UID: \"6bf989cb-53d0-4895-8bd1-377fe1451b7b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ljhws"
Jul 23 11:35:47.045423 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:47.044096    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/93649ac9-9062-46c0-adaa-1048d471211a-host-cni-netd\") pod \"ovnkube-node-nhm9p\" (UID: \"93649ac9-9062-46c0-adaa-1048d471211a\") " pod="openshift-ovn-kubernetes/ovnkube-node-nhm9p"
Jul 23 11:35:47.045423 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:47.044124    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e6553d8c-9a9e-40be-a703-77d9c5235122-etc-kubernetes\") pod \"tuned-hxr6k\" (UID: \"e6553d8c-9a9e-40be-a703-77d9c5235122\") " pod="openshift-cluster-node-tuning-operator/tuned-hxr6k"
Jul 23 11:35:47.045423 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:47.044147    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5ef3b70f-a117-4dcd-9380-5c5d36e596c0-tmp-dir\") pod \"node-resolver-dmnft\" (UID: \"5ef3b70f-a117-4dcd-9380-5c5d36e596c0\") " pod="openshift-dns/node-resolver-dmnft"
Jul 23 11:35:47.045423 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:47.044169    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0075cfcb-3377-4888-b93b-1fe1e8f4b6da-cnibin\") pod \"multus-rknqz\" (UID: \"0075cfcb-3377-4888-b93b-1fe1e8f4b6da\") " pod="openshift-multus/multus-rknqz"
Jul 23 11:35:47.045423 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:47.044195    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0075cfcb-3377-4888-b93b-1fe1e8f4b6da-host-run-k8s-cni-cncf-io\") pod \"multus-rknqz\" (UID: \"0075cfcb-3377-4888-b93b-1fe1e8f4b6da\") " pod="openshift-multus/multus-rknqz"
Jul 23 11:35:47.045423 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:47.044219    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0075cfcb-3377-4888-b93b-1fe1e8f4b6da-host-var-lib-cni-multus\") pod \"multus-rknqz\" (UID: \"0075cfcb-3377-4888-b93b-1fe1e8f4b6da\") " pod="openshift-multus/multus-rknqz"
Jul 23 11:35:47.045423 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:47.044244    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0075cfcb-3377-4888-b93b-1fe1e8f4b6da-host-run-multus-certs\") pod \"multus-rknqz\" (UID: \"0075cfcb-3377-4888-b93b-1fe1e8f4b6da\") " pod="openshift-multus/multus-rknqz"
Jul 23 11:35:47.045423 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:47.044268    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0075cfcb-3377-4888-b93b-1fe1e8f4b6da-etc-kubernetes\") pod \"multus-rknqz\" (UID: \"0075cfcb-3377-4888-b93b-1fe1e8f4b6da\") " pod="openshift-multus/multus-rknqz"
Jul 23 11:35:47.045423 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:47.044296    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-528jd\" (UniqueName: \"kubernetes.io/projected/b30ea3bc-3059-4d68-934c-32cde3d06fbb-kube-api-access-528jd\") pod \"network-check-target-l6jwv\" (UID: \"b30ea3bc-3059-4d68-934c-32cde3d06fbb\") " pod="openshift-network-diagnostics/network-check-target-l6jwv"
Jul 23 11:35:47.045423 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:47.044324    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0bcf7e84-67a4-4cf6-bb1e-016bb38f4ccb-iptables-alerter-script\") pod \"iptables-alerter-cb7gq\" (UID: \"0bcf7e84-67a4-4cf6-bb1e-016bb38f4ccb\") " pod="openshift-network-operator/iptables-alerter-cb7gq"
Jul 23 11:35:47.045423 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:47.044350    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/93649ac9-9062-46c0-adaa-1048d471211a-etc-openvswitch\") pod \"ovnkube-node-nhm9p\" (UID: \"93649ac9-9062-46c0-adaa-1048d471211a\") " pod="openshift-ovn-kubernetes/ovnkube-node-nhm9p"
Jul 23 11:35:47.045423 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:47.044377    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/93649ac9-9062-46c0-adaa-1048d471211a-host-run-netns\") pod \"ovnkube-node-nhm9p\" (UID: \"93649ac9-9062-46c0-adaa-1048d471211a\") " pod="openshift-ovn-kubernetes/ovnkube-node-nhm9p"
Jul 23 11:35:47.045423 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:47.044392    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/93649ac9-9062-46c0-adaa-1048d471211a-etc-openvswitch\") pod \"ovnkube-node-nhm9p\" (UID: \"93649ac9-9062-46c0-adaa-1048d471211a\") " pod="openshift-ovn-kubernetes/ovnkube-node-nhm9p"
Jul 23 11:35:47.045992 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:47.044402    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/2ba42487-6ccf-4e15-93df-65f0623c41ba-serviceca\") pod \"node-ca-dg4zg\" (UID: \"2ba42487-6ccf-4e15-93df-65f0623c41ba\") " pod="openshift-image-registry/node-ca-dg4zg"
Jul 23 11:35:47.045992 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:47.044435    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/93649ac9-9062-46c0-adaa-1048d471211a-host-run-netns\") pod \"ovnkube-node-nhm9p\" (UID: \"93649ac9-9062-46c0-adaa-1048d471211a\") " pod="openshift-ovn-kubernetes/ovnkube-node-nhm9p"
Jul 23 11:35:47.045992 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:47.044460    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xn75t\" (UniqueName: \"kubernetes.io/projected/2ba42487-6ccf-4e15-93df-65f0623c41ba-kube-api-access-xn75t\") pod \"node-ca-dg4zg\" (UID: \"2ba42487-6ccf-4e15-93df-65f0623c41ba\") " pod="openshift-image-registry/node-ca-dg4zg"
Jul 23 11:35:47.045992 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:47.044485    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7a2f8e1f-040a-4e01-88a7-9909e2fbc8b3-tuning-conf-dir\") pod \"multus-additional-cni-plugins-prdbw\" (UID: \"7a2f8e1f-040a-4e01-88a7-9909e2fbc8b3\") " pod="openshift-multus/multus-additional-cni-plugins-prdbw"
Jul 23 11:35:47.045992 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:47.044508    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zxvw\" (UniqueName: \"kubernetes.io/projected/7a2f8e1f-040a-4e01-88a7-9909e2fbc8b3-kube-api-access-4zxvw\") pod \"multus-additional-cni-plugins-prdbw\" (UID: \"7a2f8e1f-040a-4e01-88a7-9909e2fbc8b3\") " pod="openshift-multus/multus-additional-cni-plugins-prdbw"
Jul 23 11:35:47.045992 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:47.044536    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0075cfcb-3377-4888-b93b-1fe1e8f4b6da-multus-socket-dir-parent\") pod \"multus-rknqz\" (UID: \"0075cfcb-3377-4888-b93b-1fe1e8f4b6da\") " pod="openshift-multus/multus-rknqz"
Jul 23 11:35:47.045992 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:47.044566    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/93649ac9-9062-46c0-adaa-1048d471211a-host-slash\") pod \"ovnkube-node-nhm9p\" (UID: \"93649ac9-9062-46c0-adaa-1048d471211a\") " pod="openshift-ovn-kubernetes/ovnkube-node-nhm9p"
Jul 23 11:35:47.045992 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:47.044591    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e6553d8c-9a9e-40be-a703-77d9c5235122-etc-sysctl-d\") pod \"tuned-hxr6k\" (UID: \"e6553d8c-9a9e-40be-a703-77d9c5235122\") " pod="openshift-cluster-node-tuning-operator/tuned-hxr6k"
Jul 23 11:35:47.045992 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:47.044614    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e6553d8c-9a9e-40be-a703-77d9c5235122-etc-systemd\") pod \"tuned-hxr6k\" (UID: \"e6553d8c-9a9e-40be-a703-77d9c5235122\") " pod="openshift-cluster-node-tuning-operator/tuned-hxr6k"
Jul 23 11:35:47.045992 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:47.044620    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/93649ac9-9062-46c0-adaa-1048d471211a-host-slash\") pod \"ovnkube-node-nhm9p\" (UID: \"93649ac9-9062-46c0-adaa-1048d471211a\") " pod="openshift-ovn-kubernetes/ovnkube-node-nhm9p"
Jul 23 11:35:47.045992 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:47.044641    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7a2f8e1f-040a-4e01-88a7-9909e2fbc8b3-system-cni-dir\") pod \"multus-additional-cni-plugins-prdbw\" (UID: \"7a2f8e1f-040a-4e01-88a7-9909e2fbc8b3\") " pod="openshift-multus/multus-additional-cni-plugins-prdbw"
Jul 23 11:35:47.045992 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:47.044675    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7a2f8e1f-040a-4e01-88a7-9909e2fbc8b3-cnibin\") pod \"multus-additional-cni-plugins-prdbw\" (UID: \"7a2f8e1f-040a-4e01-88a7-9909e2fbc8b3\") " pod="openshift-multus/multus-additional-cni-plugins-prdbw"
Jul 23 11:35:47.045992 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:47.044708    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0075cfcb-3377-4888-b93b-1fe1e8f4b6da-multus-conf-dir\") pod \"multus-rknqz\" (UID: \"0075cfcb-3377-4888-b93b-1fe1e8f4b6da\") " pod="openshift-multus/multus-rknqz"
Jul 23 11:35:47.045992 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:47.044742    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0075cfcb-3377-4888-b93b-1fe1e8f4b6da-multus-daemon-config\") pod \"multus-rknqz\" (UID: \"0075cfcb-3377-4888-b93b-1fe1e8f4b6da\") " pod="openshift-multus/multus-rknqz"
Jul 23 11:35:47.045992 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:47.044801    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cbdff1cb-47fd-4ed9-bec9-419ea8da3aaa-metrics-certs\") pod \"network-metrics-daemon-pbmtq\" (UID: \"cbdff1cb-47fd-4ed9-bec9-419ea8da3aaa\") " pod="openshift-multus/network-metrics-daemon-pbmtq"
Jul 23 11:35:47.045992 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:47.044821    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/6bf989cb-53d0-4895-8bd1-377fe1451b7b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-ljhws\" (UID: \"6bf989cb-53d0-4895-8bd1-377fe1451b7b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ljhws"
Jul 23 11:35:47.046585 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:47.044836    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/73f77fae-fdff-4232-a26a-e5749f7df186-agent-certs\") pod \"konnectivity-agent-2rs9k\" (UID: \"73f77fae-fdff-4232-a26a-e5749f7df186\") " pod="kube-system/konnectivity-agent-2rs9k"
Jul 23 11:35:47.046585 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:47.044857    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rn28c\" (UniqueName: \"kubernetes.io/projected/cbdff1cb-47fd-4ed9-bec9-419ea8da3aaa-kube-api-access-rn28c\") pod \"network-metrics-daemon-pbmtq\" (UID: \"cbdff1cb-47fd-4ed9-bec9-419ea8da3aaa\") " pod="openshift-multus/network-metrics-daemon-pbmtq"
Jul 23 11:35:47.046585 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:47.044890    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/93649ac9-9062-46c0-adaa-1048d471211a-env-overrides\") pod \"ovnkube-node-nhm9p\" (UID: \"93649ac9-9062-46c0-adaa-1048d471211a\") " pod="openshift-ovn-kubernetes/ovnkube-node-nhm9p"
Jul 23 11:35:47.046585 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:47.044910    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/6bf989cb-53d0-4895-8bd1-377fe1451b7b-device-dir\") pod \"aws-ebs-csi-driver-node-ljhws\" (UID: \"6bf989cb-53d0-4895-8bd1-377fe1451b7b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ljhws"
Jul 23 11:35:47.046585 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:47.045182    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/93649ac9-9062-46c0-adaa-1048d471211a-env-overrides\") pod \"ovnkube-node-nhm9p\" (UID: \"93649ac9-9062-46c0-adaa-1048d471211a\") " pod="openshift-ovn-kubernetes/ovnkube-node-nhm9p"
Jul 23 11:35:47.047009 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:47.046992    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/93649ac9-9062-46c0-adaa-1048d471211a-ovn-node-metrics-cert\") pod \"ovnkube-node-nhm9p\" (UID: \"93649ac9-9062-46c0-adaa-1048d471211a\") " pod="openshift-ovn-kubernetes/ovnkube-node-nhm9p"
Jul 23 11:35:47.051007 ip-10-0-141-229 kubenswrapper[2563]: E0723 11:35:47.050984    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 23 11:35:47.051117 ip-10-0-141-229 kubenswrapper[2563]: E0723 11:35:47.051014    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 23 11:35:47.051117 ip-10-0-141-229 kubenswrapper[2563]: E0723 11:35:47.051029    2563 projected.go:196] Error preparing data for projected volume kube-api-access-528jd for pod openshift-network-diagnostics/network-check-target-l6jwv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 23 11:35:47.051246 ip-10-0-141-229 kubenswrapper[2563]: E0723 11:35:47.051129    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b30ea3bc-3059-4d68-934c-32cde3d06fbb-kube-api-access-528jd podName:b30ea3bc-3059-4d68-934c-32cde3d06fbb nodeName:}" failed. No retries permitted until 2026-07-23 11:35:47.551098836 +0000 UTC m=+1.944036629 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-528jd" (UniqueName: "kubernetes.io/projected/b30ea3bc-3059-4d68-934c-32cde3d06fbb-kube-api-access-528jd") pod "network-check-target-l6jwv" (UID: "b30ea3bc-3059-4d68-934c-32cde3d06fbb") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 23 11:35:47.052207 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:47.052173    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2l6nr\" (UniqueName: \"kubernetes.io/projected/0bcf7e84-67a4-4cf6-bb1e-016bb38f4ccb-kube-api-access-2l6nr\") pod \"iptables-alerter-cb7gq\" (UID: \"0bcf7e84-67a4-4cf6-bb1e-016bb38f4ccb\") " pod="openshift-network-operator/iptables-alerter-cb7gq"
Jul 23 11:35:47.052823 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:47.052804    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxwcn\" (UniqueName: \"kubernetes.io/projected/93649ac9-9062-46c0-adaa-1048d471211a-kube-api-access-hxwcn\") pod \"ovnkube-node-nhm9p\" (UID: \"93649ac9-9062-46c0-adaa-1048d471211a\") " pod="openshift-ovn-kubernetes/ovnkube-node-nhm9p"
Jul 23 11:35:47.064301 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:47.064265    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod192fe6abc0214ce0c06c38f944481f45.slice/crio-694c07012b66221664dbb1befdf85f0273512a4a62e0ce7be45ae27a63660f01 WatchSource:0}: Error finding container 694c07012b66221664dbb1befdf85f0273512a4a62e0ce7be45ae27a63660f01: Status 404 returned error can't find the container with id 694c07012b66221664dbb1befdf85f0273512a4a62e0ce7be45ae27a63660f01
Jul 23 11:35:47.065184 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:47.065163    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92a19fc888e8076d0b3fbd4b9dc96a0a.slice/crio-c2e8e69a11a26de1fbf6c41e460855d8764ab66180b2b8863527eeedabee7962 WatchSource:0}: Error finding container c2e8e69a11a26de1fbf6c41e460855d8764ab66180b2b8863527eeedabee7962: Status 404 returned error can't find the container with id c2e8e69a11a26de1fbf6c41e460855d8764ab66180b2b8863527eeedabee7962
Jul 23 11:35:47.070586 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:47.070570    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 23 11:35:47.075606 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:47.075568    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-229.ec2.internal" event={"ID":"92a19fc888e8076d0b3fbd4b9dc96a0a","Type":"ContainerStarted","Data":"c2e8e69a11a26de1fbf6c41e460855d8764ab66180b2b8863527eeedabee7962"}
Jul 23 11:35:47.076701 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:47.076677    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-229.ec2.internal" event={"ID":"192fe6abc0214ce0c06c38f944481f45","Type":"ContainerStarted","Data":"694c07012b66221664dbb1befdf85f0273512a4a62e0ce7be45ae27a63660f01"}
Jul 23 11:35:47.135605 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:47.135578    2563 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 23 11:35:47.145254 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:47.145228    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/6bf989cb-53d0-4895-8bd1-377fe1451b7b-registration-dir\") pod \"aws-ebs-csi-driver-node-ljhws\" (UID: \"6bf989cb-53d0-4895-8bd1-377fe1451b7b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ljhws"
Jul 23 11:35:47.145387 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:47.145276    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/6bf989cb-53d0-4895-8bd1-377fe1451b7b-etc-selinux\") pod \"aws-ebs-csi-driver-node-ljhws\" (UID: \"6bf989cb-53d0-4895-8bd1-377fe1451b7b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ljhws"
Jul 23 11:35:47.145387 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:47.145298    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e6553d8c-9a9e-40be-a703-77d9c5235122-etc-sysconfig\") pod \"tuned-hxr6k\" (UID: \"e6553d8c-9a9e-40be-a703-77d9c5235122\") " pod="openshift-cluster-node-tuning-operator/tuned-hxr6k"
Jul 23 11:35:47.145387 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:47.145315    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/6bf989cb-53d0-4895-8bd1-377fe1451b7b-sys-fs\") pod \"aws-ebs-csi-driver-node-ljhws\" (UID: \"6bf989cb-53d0-4895-8bd1-377fe1451b7b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ljhws"
Jul 23 11:35:47.145387 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:47.145357    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/6bf989cb-53d0-4895-8bd1-377fe1451b7b-registration-dir\") pod \"aws-ebs-csi-driver-node-ljhws\" (UID: \"6bf989cb-53d0-4895-8bd1-377fe1451b7b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ljhws"
Jul 23 11:35:47.145387 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:47.145383    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/6bf989cb-53d0-4895-8bd1-377fe1451b7b-sys-fs\") pod \"aws-ebs-csi-driver-node-ljhws\" (UID: \"6bf989cb-53d0-4895-8bd1-377fe1451b7b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ljhws"
Jul 23 11:35:47.145620 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:47.145385    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e6553d8c-9a9e-40be-a703-77d9c5235122-etc-sysconfig\") pod \"tuned-hxr6k\" (UID: \"e6553d8c-9a9e-40be-a703-77d9c5235122\") " pod="openshift-cluster-node-tuning-operator/tuned-hxr6k"
Jul 23 11:35:47.145620 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:47.145386    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e6553d8c-9a9e-40be-a703-77d9c5235122-etc-kubernetes\") pod \"tuned-hxr6k\" (UID: \"e6553d8c-9a9e-40be-a703-77d9c5235122\") " pod="openshift-cluster-node-tuning-operator/tuned-hxr6k"
Jul 23 11:35:47.145620 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:47.145427    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/6bf989cb-53d0-4895-8bd1-377fe1451b7b-etc-selinux\") pod \"aws-ebs-csi-driver-node-ljhws\" (UID: \"6bf989cb-53d0-4895-8bd1-377fe1451b7b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ljhws"
Jul 23 11:35:47.145620 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:47.145444    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5ef3b70f-a117-4dcd-9380-5c5d36e596c0-tmp-dir\") pod \"node-resolver-dmnft\" (UID: \"5ef3b70f-a117-4dcd-9380-5c5d36e596c0\") " pod="openshift-dns/node-resolver-dmnft"
Jul 23 11:35:47.145620 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:47.145469    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0075cfcb-3377-4888-b93b-1fe1e8f4b6da-cnibin\") pod \"multus-rknqz\" (UID: \"0075cfcb-3377-4888-b93b-1fe1e8f4b6da\") " pod="openshift-multus/multus-rknqz"
Jul 23 11:35:47.145620 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:47.145485    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0075cfcb-3377-4888-b93b-1fe1e8f4b6da-host-run-k8s-cni-cncf-io\") pod \"multus-rknqz\" (UID: \"0075cfcb-3377-4888-b93b-1fe1e8f4b6da\") " pod="openshift-multus/multus-rknqz"
Jul 23 11:35:47.145620 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:47.145501    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0075cfcb-3377-4888-b93b-1fe1e8f4b6da-host-var-lib-cni-multus\") pod \"multus-rknqz\" (UID: \"0075cfcb-3377-4888-b93b-1fe1e8f4b6da\") " pod="openshift-multus/multus-rknqz"
Jul 23 11:35:47.145620 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:47.145526    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0075cfcb-3377-4888-b93b-1fe1e8f4b6da-host-run-multus-certs\") pod \"multus-rknqz\" (UID: \"0075cfcb-3377-4888-b93b-1fe1e8f4b6da\") " pod="openshift-multus/multus-rknqz"
Jul 23 11:35:47.145620 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:47.145545    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0075cfcb-3377-4888-b93b-1fe1e8f4b6da-cnibin\") pod \"multus-rknqz\" (UID: \"0075cfcb-3377-4888-b93b-1fe1e8f4b6da\") " pod="openshift-multus/multus-rknqz"
Jul 23 11:35:47.145620 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:47.145445    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e6553d8c-9a9e-40be-a703-77d9c5235122-etc-kubernetes\") pod \"tuned-hxr6k\" (UID: \"e6553d8c-9a9e-40be-a703-77d9c5235122\") " pod="openshift-cluster-node-tuning-operator/tuned-hxr6k"
Jul 23 11:35:47.145620 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:47.145568    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0075cfcb-3377-4888-b93b-1fe1e8f4b6da-host-run-multus-certs\") pod \"multus-rknqz\" (UID: \"0075cfcb-3377-4888-b93b-1fe1e8f4b6da\") " pod="openshift-multus/multus-rknqz"
Jul 23 11:35:47.145620 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:47.145549    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0075cfcb-3377-4888-b93b-1fe1e8f4b6da-host-run-k8s-cni-cncf-io\") pod \"multus-rknqz\" (UID: \"0075cfcb-3377-4888-b93b-1fe1e8f4b6da\") " pod="openshift-multus/multus-rknqz"
Jul 23 11:35:47.145620 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:47.145571    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0075cfcb-3377-4888-b93b-1fe1e8f4b6da-host-var-lib-cni-multus\") pod \"multus-rknqz\" (UID: \"0075cfcb-3377-4888-b93b-1fe1e8f4b6da\") " pod="openshift-multus/multus-rknqz"
Jul 23 11:35:47.145620 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:47.145588    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0075cfcb-3377-4888-b93b-1fe1e8f4b6da-etc-kubernetes\") pod \"multus-rknqz\" (UID: \"0075cfcb-3377-4888-b93b-1fe1e8f4b6da\") " pod="openshift-multus/multus-rknqz"
Jul 23 11:35:47.145620 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:47.145618    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/2ba42487-6ccf-4e15-93df-65f0623c41ba-serviceca\") pod \"node-ca-dg4zg\" (UID: \"2ba42487-6ccf-4e15-93df-65f0623c41ba\") " pod="openshift-image-registry/node-ca-dg4zg"
Jul 23 11:35:47.146361 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:47.145643    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xn75t\" (UniqueName: \"kubernetes.io/projected/2ba42487-6ccf-4e15-93df-65f0623c41ba-kube-api-access-xn75t\") pod \"node-ca-dg4zg\" (UID: \"2ba42487-6ccf-4e15-93df-65f0623c41ba\") " pod="openshift-image-registry/node-ca-dg4zg"
Jul 23 11:35:47.146361 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:47.145663    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7a2f8e1f-040a-4e01-88a7-9909e2fbc8b3-tuning-conf-dir\") pod \"multus-additional-cni-plugins-prdbw\" (UID: \"7a2f8e1f-040a-4e01-88a7-9909e2fbc8b3\") " pod="openshift-multus/multus-additional-cni-plugins-prdbw"
Jul 23 11:35:47.146361 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:47.145652    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0075cfcb-3377-4888-b93b-1fe1e8f4b6da-etc-kubernetes\") pod \"multus-rknqz\" (UID: \"0075cfcb-3377-4888-b93b-1fe1e8f4b6da\") " pod="openshift-multus/multus-rknqz"
Jul 23 11:35:47.146361 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:47.145682    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zxvw\" (UniqueName: \"kubernetes.io/projected/7a2f8e1f-040a-4e01-88a7-9909e2fbc8b3-kube-api-access-4zxvw\") pod \"multus-additional-cni-plugins-prdbw\" (UID: \"7a2f8e1f-040a-4e01-88a7-9909e2fbc8b3\") " pod="openshift-multus/multus-additional-cni-plugins-prdbw"
Jul 23 11:35:47.146361 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:47.145725    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0075cfcb-3377-4888-b93b-1fe1e8f4b6da-multus-socket-dir-parent\") pod \"multus-rknqz\" (UID: \"0075cfcb-3377-4888-b93b-1fe1e8f4b6da\") " pod="openshift-multus/multus-rknqz"
Jul 23 11:35:47.146361 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:47.145751    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e6553d8c-9a9e-40be-a703-77d9c5235122-etc-sysctl-d\") pod \"tuned-hxr6k\" (UID: \"e6553d8c-9a9e-40be-a703-77d9c5235122\") " pod="openshift-cluster-node-tuning-operator/tuned-hxr6k"
Jul 23 11:35:47.146361 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:47.145779    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e6553d8c-9a9e-40be-a703-77d9c5235122-etc-systemd\") pod \"tuned-hxr6k\" (UID: \"e6553d8c-9a9e-40be-a703-77d9c5235122\") " pod="openshift-cluster-node-tuning-operator/tuned-hxr6k"
Jul 23 11:35:47.146361 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:47.145802    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7a2f8e1f-040a-4e01-88a7-9909e2fbc8b3-system-cni-dir\") pod \"multus-additional-cni-plugins-prdbw\" (UID: \"7a2f8e1f-040a-4e01-88a7-9909e2fbc8b3\") " pod="openshift-multus/multus-additional-cni-plugins-prdbw"
Jul 23 11:35:47.146361 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:47.145835    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7a2f8e1f-040a-4e01-88a7-9909e2fbc8b3-cnibin\") pod \"multus-additional-cni-plugins-prdbw\" (UID: \"7a2f8e1f-040a-4e01-88a7-9909e2fbc8b3\") " pod="openshift-multus/multus-additional-cni-plugins-prdbw"
Jul 23 11:35:47.146361 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:47.145841    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7a2f8e1f-040a-4e01-88a7-9909e2fbc8b3-tuning-conf-dir\") pod \"multus-additional-cni-plugins-prdbw\" (UID: \"7a2f8e1f-040a-4e01-88a7-9909e2fbc8b3\") " pod="openshift-multus/multus-additional-cni-plugins-prdbw"
Jul 23 11:35:47.146361 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:47.145862    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0075cfcb-3377-4888-b93b-1fe1e8f4b6da-multus-conf-dir\") pod \"multus-rknqz\" (UID: \"0075cfcb-3377-4888-b93b-1fe1e8f4b6da\") " pod="openshift-multus/multus-rknqz"
Jul 23 11:35:47.146361 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:47.145874    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0075cfcb-3377-4888-b93b-1fe1e8f4b6da-multus-socket-dir-parent\") pod \"multus-rknqz\" (UID: \"0075cfcb-3377-4888-b93b-1fe1e8f4b6da\") " pod="openshift-multus/multus-rknqz"
Jul 23 11:35:47.146361 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:47.145889    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0075cfcb-3377-4888-b93b-1fe1e8f4b6da-multus-daemon-config\") pod \"multus-rknqz\" (UID: \"0075cfcb-3377-4888-b93b-1fe1e8f4b6da\") " pod="openshift-multus/multus-rknqz"
Jul 23 11:35:47.146361 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:47.145894    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e6553d8c-9a9e-40be-a703-77d9c5235122-etc-systemd\") pod \"tuned-hxr6k\" (UID: \"e6553d8c-9a9e-40be-a703-77d9c5235122\") " pod="openshift-cluster-node-tuning-operator/tuned-hxr6k"
Jul 23 11:35:47.146361 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:47.145901    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7a2f8e1f-040a-4e01-88a7-9909e2fbc8b3-system-cni-dir\") pod \"multus-additional-cni-plugins-prdbw\" (UID: \"7a2f8e1f-040a-4e01-88a7-9909e2fbc8b3\") " pod="openshift-multus/multus-additional-cni-plugins-prdbw"
Jul 23 11:35:47.146361 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:47.145915    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cbdff1cb-47fd-4ed9-bec9-419ea8da3aaa-metrics-certs\") pod \"network-metrics-daemon-pbmtq\" (UID: \"cbdff1cb-47fd-4ed9-bec9-419ea8da3aaa\") " pod="openshift-multus/network-metrics-daemon-pbmtq"
Jul 23 11:35:47.146361 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:47.145925    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7a2f8e1f-040a-4e01-88a7-9909e2fbc8b3-cnibin\") pod \"multus-additional-cni-plugins-prdbw\" (UID: \"7a2f8e1f-040a-4e01-88a7-9909e2fbc8b3\") " pod="openshift-multus/multus-additional-cni-plugins-prdbw"
Jul 23 11:35:47.147284 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:47.145942    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/6bf989cb-53d0-4895-8bd1-377fe1451b7b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-ljhws\" (UID: \"6bf989cb-53d0-4895-8bd1-377fe1451b7b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ljhws"
Jul 23 11:35:47.147284 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:47.145949    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0075cfcb-3377-4888-b93b-1fe1e8f4b6da-multus-conf-dir\") pod \"multus-rknqz\" (UID: \"0075cfcb-3377-4888-b93b-1fe1e8f4b6da\") " pod="openshift-multus/multus-rknqz"
Jul 23 11:35:47.147284 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:47.145941    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e6553d8c-9a9e-40be-a703-77d9c5235122-etc-sysctl-d\") pod \"tuned-hxr6k\" (UID: \"e6553d8c-9a9e-40be-a703-77d9c5235122\") " pod="openshift-cluster-node-tuning-operator/tuned-hxr6k"
Jul 23 11:35:47.147284 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:47.145968    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/73f77fae-fdff-4232-a26a-e5749f7df186-agent-certs\") pod \"konnectivity-agent-2rs9k\" (UID: \"73f77fae-fdff-4232-a26a-e5749f7df186\") " pod="kube-system/konnectivity-agent-2rs9k"
Jul 23 11:35:47.147284 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:47.145994    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rn28c\" (UniqueName: \"kubernetes.io/projected/cbdff1cb-47fd-4ed9-bec9-419ea8da3aaa-kube-api-access-rn28c\") pod \"network-metrics-daemon-pbmtq\" (UID: \"cbdff1cb-47fd-4ed9-bec9-419ea8da3aaa\") " pod="openshift-multus/network-metrics-daemon-pbmtq"
Jul 23 11:35:47.147284 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:47.146003    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/6bf989cb-53d0-4895-8bd1-377fe1451b7b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-ljhws\" (UID: \"6bf989cb-53d0-4895-8bd1-377fe1451b7b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ljhws"
Jul 23 11:35:47.147284 ip-10-0-141-229 kubenswrapper[2563]: E0723 11:35:47.146009    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 23 11:35:47.147284 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:47.146075    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/6bf989cb-53d0-4895-8bd1-377fe1451b7b-device-dir\") pod \"aws-ebs-csi-driver-node-ljhws\" (UID: \"6bf989cb-53d0-4895-8bd1-377fe1451b7b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ljhws"
Jul 23 11:35:47.147284 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:47.146022    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/6bf989cb-53d0-4895-8bd1-377fe1451b7b-device-dir\") pod \"aws-ebs-csi-driver-node-ljhws\" (UID: \"6bf989cb-53d0-4895-8bd1-377fe1451b7b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ljhws"
Jul 23 11:35:47.147284 ip-10-0-141-229 kubenswrapper[2563]: E0723 11:35:47.146098    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cbdff1cb-47fd-4ed9-bec9-419ea8da3aaa-metrics-certs podName:cbdff1cb-47fd-4ed9-bec9-419ea8da3aaa nodeName:}" failed. No retries permitted until 2026-07-23 11:35:47.646076021 +0000 UTC m=+2.039013807 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cbdff1cb-47fd-4ed9-bec9-419ea8da3aaa-metrics-certs") pod "network-metrics-daemon-pbmtq" (UID: "cbdff1cb-47fd-4ed9-bec9-419ea8da3aaa") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 23 11:35:47.147284 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:47.146188    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/2ba42487-6ccf-4e15-93df-65f0623c41ba-serviceca\") pod \"node-ca-dg4zg\" (UID: \"2ba42487-6ccf-4e15-93df-65f0623c41ba\") " pod="openshift-image-registry/node-ca-dg4zg"
Jul 23 11:35:47.147284 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:47.146191    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e6553d8c-9a9e-40be-a703-77d9c5235122-var-lib-kubelet\") pod \"tuned-hxr6k\" (UID: \"e6553d8c-9a9e-40be-a703-77d9c5235122\") " pod="openshift-cluster-node-tuning-operator/tuned-hxr6k"
Jul 23 11:35:47.147284 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:47.145748    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5ef3b70f-a117-4dcd-9380-5c5d36e596c0-tmp-dir\") pod \"node-resolver-dmnft\" (UID: \"5ef3b70f-a117-4dcd-9380-5c5d36e596c0\") " pod="openshift-dns/node-resolver-dmnft"
Jul 23 11:35:47.147284 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:47.146221    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/7a0946cf-0f04-49d5-ab74-6f64a48f8e5d-kubelet-config\") pod \"global-pull-secret-syncer-cvbbt\" (UID: \"7a0946cf-0f04-49d5-ab74-6f64a48f8e5d\") " pod="kube-system/global-pull-secret-syncer-cvbbt"
Jul 23 11:35:47.147284 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:47.146249    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e6553d8c-9a9e-40be-a703-77d9c5235122-var-lib-kubelet\") pod \"tuned-hxr6k\" (UID: \"e6553d8c-9a9e-40be-a703-77d9c5235122\") " pod="openshift-cluster-node-tuning-operator/tuned-hxr6k"
Jul 23 11:35:47.147284 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:47.146277    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/7a0946cf-0f04-49d5-ab74-6f64a48f8e5d-kubelet-config\") pod \"global-pull-secret-syncer-cvbbt\" (UID: \"7a0946cf-0f04-49d5-ab74-6f64a48f8e5d\") " pod="kube-system/global-pull-secret-syncer-cvbbt"
Jul 23 11:35:47.147284 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:47.146297    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/7a0946cf-0f04-49d5-ab74-6f64a48f8e5d-dbus\") pod \"global-pull-secret-syncer-cvbbt\" (UID: \"7a0946cf-0f04-49d5-ab74-6f64a48f8e5d\") " pod="kube-system/global-pull-secret-syncer-cvbbt"
Jul 23 11:35:47.148094 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:47.146328    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e6553d8c-9a9e-40be-a703-77d9c5235122-run\") pod \"tuned-hxr6k\" (UID: \"e6553d8c-9a9e-40be-a703-77d9c5235122\") " pod="openshift-cluster-node-tuning-operator/tuned-hxr6k"
Jul 23 11:35:47.148094 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:47.146376    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e6553d8c-9a9e-40be-a703-77d9c5235122-host\") pod \"tuned-hxr6k\" (UID: \"e6553d8c-9a9e-40be-a703-77d9c5235122\") " pod="openshift-cluster-node-tuning-operator/tuned-hxr6k"
Jul 23 11:35:47.148094 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:47.146399    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e6553d8c-9a9e-40be-a703-77d9c5235122-run\") pod \"tuned-hxr6k\" (UID: \"e6553d8c-9a9e-40be-a703-77d9c5235122\") " pod="openshift-cluster-node-tuning-operator/tuned-hxr6k"
Jul 23 11:35:47.148094 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:47.146403    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-blmdc\" (UniqueName: \"kubernetes.io/projected/e6553d8c-9a9e-40be-a703-77d9c5235122-kube-api-access-blmdc\") pod \"tuned-hxr6k\" (UID: \"e6553d8c-9a9e-40be-a703-77d9c5235122\") " pod="openshift-cluster-node-tuning-operator/tuned-hxr6k"
Jul 23 11:35:47.148094 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:47.146434    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/7a0946cf-0f04-49d5-ab74-6f64a48f8e5d-dbus\") pod \"global-pull-secret-syncer-cvbbt\" (UID: \"7a0946cf-0f04-49d5-ab74-6f64a48f8e5d\") " pod="kube-system/global-pull-secret-syncer-cvbbt"
Jul 23 11:35:47.148094 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:47.146438    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0075cfcb-3377-4888-b93b-1fe1e8f4b6da-hostroot\") pod \"multus-rknqz\" (UID: \"0075cfcb-3377-4888-b93b-1fe1e8f4b6da\") " pod="openshift-multus/multus-rknqz"
Jul 23 11:35:47.148094 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:47.146449    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0075cfcb-3377-4888-b93b-1fe1e8f4b6da-multus-daemon-config\") pod \"multus-rknqz\" (UID: \"0075cfcb-3377-4888-b93b-1fe1e8f4b6da\") " pod="openshift-multus/multus-rknqz"
Jul 23 11:35:47.148094 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:47.146467    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0075cfcb-3377-4888-b93b-1fe1e8f4b6da-hostroot\") pod \"multus-rknqz\" (UID: \"0075cfcb-3377-4888-b93b-1fe1e8f4b6da\") " pod="openshift-multus/multus-rknqz"
Jul 23 11:35:47.148094 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:47.146488    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/7a2f8e1f-040a-4e01-88a7-9909e2fbc8b3-os-release\") pod \"multus-additional-cni-plugins-prdbw\" (UID: \"7a2f8e1f-040a-4e01-88a7-9909e2fbc8b3\") " pod="openshift-multus/multus-additional-cni-plugins-prdbw"
Jul 23 11:35:47.148094 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:47.146513    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0075cfcb-3377-4888-b93b-1fe1e8f4b6da-multus-cni-dir\") pod \"multus-rknqz\" (UID: \"0075cfcb-3377-4888-b93b-1fe1e8f4b6da\") " pod="openshift-multus/multus-rknqz"
Jul 23 11:35:47.148094 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:47.146522    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e6553d8c-9a9e-40be-a703-77d9c5235122-host\") pod \"tuned-hxr6k\" (UID: \"e6553d8c-9a9e-40be-a703-77d9c5235122\") " pod="openshift-cluster-node-tuning-operator/tuned-hxr6k"
Jul 23 11:35:47.148094 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:47.146534    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0075cfcb-3377-4888-b93b-1fe1e8f4b6da-host-var-lib-cni-bin\") pod \"multus-rknqz\" (UID: \"0075cfcb-3377-4888-b93b-1fe1e8f4b6da\") " pod="openshift-multus/multus-rknqz"
Jul 23 11:35:47.148094 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:47.146571    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxpkb\" (UniqueName: \"kubernetes.io/projected/0075cfcb-3377-4888-b93b-1fe1e8f4b6da-kube-api-access-qxpkb\") pod \"multus-rknqz\" (UID: \"0075cfcb-3377-4888-b93b-1fe1e8f4b6da\") " pod="openshift-multus/multus-rknqz"
Jul 23 11:35:47.148094 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:47.146585    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0075cfcb-3377-4888-b93b-1fe1e8f4b6da-host-var-lib-cni-bin\") pod \"multus-rknqz\" (UID: \"0075cfcb-3377-4888-b93b-1fe1e8f4b6da\") " pod="openshift-multus/multus-rknqz"
Jul 23 11:35:47.148094 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:47.146592    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e6553d8c-9a9e-40be-a703-77d9c5235122-sys\") pod \"tuned-hxr6k\" (UID: \"e6553d8c-9a9e-40be-a703-77d9c5235122\") " pod="openshift-cluster-node-tuning-operator/tuned-hxr6k"
Jul 23 11:35:47.148094 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:47.146605    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0075cfcb-3377-4888-b93b-1fe1e8f4b6da-multus-cni-dir\") pod \"multus-rknqz\" (UID: \"0075cfcb-3377-4888-b93b-1fe1e8f4b6da\") " pod="openshift-multus/multus-rknqz"
Jul 23 11:35:47.148094 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:47.146636    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e6553d8c-9a9e-40be-a703-77d9c5235122-lib-modules\") pod \"tuned-hxr6k\" (UID: \"e6553d8c-9a9e-40be-a703-77d9c5235122\") " pod="openshift-cluster-node-tuning-operator/tuned-hxr6k"
Jul 23 11:35:47.148094 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:47.146637    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e6553d8c-9a9e-40be-a703-77d9c5235122-sys\") pod \"tuned-hxr6k\" (UID: \"e6553d8c-9a9e-40be-a703-77d9c5235122\") " pod="openshift-cluster-node-tuning-operator/tuned-hxr6k"
Jul 23 11:35:47.148679 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:47.146641    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/7a2f8e1f-040a-4e01-88a7-9909e2fbc8b3-os-release\") pod \"multus-additional-cni-plugins-prdbw\" (UID: \"7a2f8e1f-040a-4e01-88a7-9909e2fbc8b3\") " pod="openshift-multus/multus-additional-cni-plugins-prdbw"
Jul 23 11:35:47.148679 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:47.146669    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e6553d8c-9a9e-40be-a703-77d9c5235122-tmp\") pod \"tuned-hxr6k\" (UID: \"e6553d8c-9a9e-40be-a703-77d9c5235122\") " pod="openshift-cluster-node-tuning-operator/tuned-hxr6k"
Jul 23 11:35:47.148679 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:47.146694    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2ba42487-6ccf-4e15-93df-65f0623c41ba-host\") pod \"node-ca-dg4zg\" (UID: \"2ba42487-6ccf-4e15-93df-65f0623c41ba\") " pod="openshift-image-registry/node-ca-dg4zg"
Jul 23 11:35:47.148679 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:47.146738    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e6553d8c-9a9e-40be-a703-77d9c5235122-lib-modules\") pod \"tuned-hxr6k\" (UID: \"e6553d8c-9a9e-40be-a703-77d9c5235122\") " pod="openshift-cluster-node-tuning-operator/tuned-hxr6k"
Jul 23 11:35:47.148679 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:47.146756    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0075cfcb-3377-4888-b93b-1fe1e8f4b6da-host-var-lib-kubelet\") pod \"multus-rknqz\" (UID: \"0075cfcb-3377-4888-b93b-1fe1e8f4b6da\") " pod="openshift-multus/multus-rknqz"
Jul 23 11:35:47.148679 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:47.146787    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e6553d8c-9a9e-40be-a703-77d9c5235122-etc-modprobe-d\") pod \"tuned-hxr6k\" (UID: \"e6553d8c-9a9e-40be-a703-77d9c5235122\") " pod="openshift-cluster-node-tuning-operator/tuned-hxr6k"
Jul 23 11:35:47.148679 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:47.146814    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s75l6\" (UniqueName: \"kubernetes.io/projected/6bf989cb-53d0-4895-8bd1-377fe1451b7b-kube-api-access-s75l6\") pod \"aws-ebs-csi-driver-node-ljhws\" (UID: \"6bf989cb-53d0-4895-8bd1-377fe1451b7b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ljhws"
Jul 23 11:35:47.148679 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:47.146757    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2ba42487-6ccf-4e15-93df-65f0623c41ba-host\") pod \"node-ca-dg4zg\" (UID: \"2ba42487-6ccf-4e15-93df-65f0623c41ba\") " pod="openshift-image-registry/node-ca-dg4zg"
Jul 23 11:35:47.148679 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:47.146838    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0075cfcb-3377-4888-b93b-1fe1e8f4b6da-host-run-netns\") pod \"multus-rknqz\" (UID: \"0075cfcb-3377-4888-b93b-1fe1e8f4b6da\") " pod="openshift-multus/multus-rknqz"
Jul 23 11:35:47.148679 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:47.146844    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0075cfcb-3377-4888-b93b-1fe1e8f4b6da-host-var-lib-kubelet\") pod \"multus-rknqz\" (UID: \"0075cfcb-3377-4888-b93b-1fe1e8f4b6da\") " pod="openshift-multus/multus-rknqz"
Jul 23 11:35:47.148679 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:47.146871    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7a0946cf-0f04-49d5-ab74-6f64a48f8e5d-original-pull-secret\") pod \"global-pull-secret-syncer-cvbbt\" (UID: \"7a0946cf-0f04-49d5-ab74-6f64a48f8e5d\") " pod="kube-system/global-pull-secret-syncer-cvbbt"
Jul 23 11:35:47.148679 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:47.146892    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0075cfcb-3377-4888-b93b-1fe1e8f4b6da-host-run-netns\") pod \"multus-rknqz\" (UID: \"0075cfcb-3377-4888-b93b-1fe1e8f4b6da\") " pod="openshift-multus/multus-rknqz"
Jul 23 11:35:47.148679 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:47.146898    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7a2f8e1f-040a-4e01-88a7-9909e2fbc8b3-cni-binary-copy\") pod \"multus-additional-cni-plugins-prdbw\" (UID: \"7a2f8e1f-040a-4e01-88a7-9909e2fbc8b3\") " pod="openshift-multus/multus-additional-cni-plugins-prdbw"
Jul 23 11:35:47.148679 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:47.146925    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0075cfcb-3377-4888-b93b-1fe1e8f4b6da-system-cni-dir\") pod \"multus-rknqz\" (UID: \"0075cfcb-3377-4888-b93b-1fe1e8f4b6da\") " pod="openshift-multus/multus-rknqz"
Jul 23 11:35:47.148679 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:47.146954    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/6bf989cb-53d0-4895-8bd1-377fe1451b7b-socket-dir\") pod \"aws-ebs-csi-driver-node-ljhws\" (UID: \"6bf989cb-53d0-4895-8bd1-377fe1451b7b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ljhws"
Jul 23 11:35:47.148679 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:47.146979    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8kc6s\" (UniqueName: \"kubernetes.io/projected/5ef3b70f-a117-4dcd-9380-5c5d36e596c0-kube-api-access-8kc6s\") pod \"node-resolver-dmnft\" (UID: \"5ef3b70f-a117-4dcd-9380-5c5d36e596c0\") " pod="openshift-dns/node-resolver-dmnft"
Jul 23 11:35:47.148679 ip-10-0-141-229 kubenswrapper[2563]: E0723 11:35:47.146972    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 23 11:35:47.148679 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:47.147035    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0075cfcb-3377-4888-b93b-1fe1e8f4b6da-system-cni-dir\") pod \"multus-rknqz\" (UID: \"0075cfcb-3377-4888-b93b-1fe1e8f4b6da\") " pod="openshift-multus/multus-rknqz"
Jul 23 11:35:47.149234 ip-10-0-141-229 kubenswrapper[2563]: E0723 11:35:47.147053    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7a0946cf-0f04-49d5-ab74-6f64a48f8e5d-original-pull-secret podName:7a0946cf-0f04-49d5-ab74-6f64a48f8e5d nodeName:}" failed. No retries permitted until 2026-07-23 11:35:47.647035063 +0000 UTC m=+2.039972830 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7a0946cf-0f04-49d5-ab74-6f64a48f8e5d-original-pull-secret") pod "global-pull-secret-syncer-cvbbt" (UID: "7a0946cf-0f04-49d5-ab74-6f64a48f8e5d") : object "kube-system"/"original-pull-secret" not registered
Jul 23 11:35:47.149234 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:47.147072    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/73f77fae-fdff-4232-a26a-e5749f7df186-konnectivity-ca\") pod \"konnectivity-agent-2rs9k\" (UID: \"73f77fae-fdff-4232-a26a-e5749f7df186\") " pod="kube-system/konnectivity-agent-2rs9k"
Jul 23 11:35:47.149234 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:47.147105    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/7a2f8e1f-040a-4e01-88a7-9909e2fbc8b3-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-prdbw\" (UID: \"7a2f8e1f-040a-4e01-88a7-9909e2fbc8b3\") " pod="openshift-multus/multus-additional-cni-plugins-prdbw"
Jul 23 11:35:47.149234 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:47.147132    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e6553d8c-9a9e-40be-a703-77d9c5235122-etc-tuned\") pod \"tuned-hxr6k\" (UID: \"e6553d8c-9a9e-40be-a703-77d9c5235122\") " pod="openshift-cluster-node-tuning-operator/tuned-hxr6k"
Jul 23 11:35:47.149234 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:47.147146    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/6bf989cb-53d0-4895-8bd1-377fe1451b7b-socket-dir\") pod \"aws-ebs-csi-driver-node-ljhws\" (UID: \"6bf989cb-53d0-4895-8bd1-377fe1451b7b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ljhws"
Jul 23 11:35:47.149234 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:47.147157    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/5ef3b70f-a117-4dcd-9380-5c5d36e596c0-hosts-file\") pod \"node-resolver-dmnft\" (UID: \"5ef3b70f-a117-4dcd-9380-5c5d36e596c0\") " pod="openshift-dns/node-resolver-dmnft"
Jul 23 11:35:47.149234 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:47.147196    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/0075cfcb-3377-4888-b93b-1fe1e8f4b6da-os-release\") pod \"multus-rknqz\" (UID: \"0075cfcb-3377-4888-b93b-1fe1e8f4b6da\") " pod="openshift-multus/multus-rknqz"
Jul 23 11:35:47.149234 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:47.147204    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/5ef3b70f-a117-4dcd-9380-5c5d36e596c0-hosts-file\") pod \"node-resolver-dmnft\" (UID: \"5ef3b70f-a117-4dcd-9380-5c5d36e596c0\") " pod="openshift-dns/node-resolver-dmnft"
Jul 23 11:35:47.149234 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:47.147228    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e6553d8c-9a9e-40be-a703-77d9c5235122-etc-sysctl-conf\") pod \"tuned-hxr6k\" (UID: \"e6553d8c-9a9e-40be-a703-77d9c5235122\") " pod="openshift-cluster-node-tuning-operator/tuned-hxr6k"
Jul 23 11:35:47.149234 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:47.147294    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/0075cfcb-3377-4888-b93b-1fe1e8f4b6da-os-release\") pod \"multus-rknqz\" (UID: \"0075cfcb-3377-4888-b93b-1fe1e8f4b6da\") " pod="openshift-multus/multus-rknqz"
Jul 23 11:35:47.149234 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:47.147433    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e6553d8c-9a9e-40be-a703-77d9c5235122-etc-sysctl-conf\") pod \"tuned-hxr6k\" (UID: \"e6553d8c-9a9e-40be-a703-77d9c5235122\") " pod="openshift-cluster-node-tuning-operator/tuned-hxr6k"
Jul 23 11:35:47.149234 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:47.147471    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/7a2f8e1f-040a-4e01-88a7-9909e2fbc8b3-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-prdbw\" (UID: \"7a2f8e1f-040a-4e01-88a7-9909e2fbc8b3\") " pod="openshift-multus/multus-additional-cni-plugins-prdbw"
Jul 23 11:35:47.149234 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:47.147506    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0075cfcb-3377-4888-b93b-1fe1e8f4b6da-cni-binary-copy\") pod \"multus-rknqz\" (UID: \"0075cfcb-3377-4888-b93b-1fe1e8f4b6da\") " pod="openshift-multus/multus-rknqz"
Jul 23 11:35:47.149234 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:47.147520    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7a2f8e1f-040a-4e01-88a7-9909e2fbc8b3-cni-binary-copy\") pod \"multus-additional-cni-plugins-prdbw\" (UID: \"7a2f8e1f-040a-4e01-88a7-9909e2fbc8b3\") " pod="openshift-multus/multus-additional-cni-plugins-prdbw"
Jul 23 11:35:47.149234 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:47.147543    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/73f77fae-fdff-4232-a26a-e5749f7df186-konnectivity-ca\") pod \"konnectivity-agent-2rs9k\" (UID: \"73f77fae-fdff-4232-a26a-e5749f7df186\") " pod="kube-system/konnectivity-agent-2rs9k"
Jul 23 11:35:47.149234 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:47.147624    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e6553d8c-9a9e-40be-a703-77d9c5235122-etc-modprobe-d\") pod \"tuned-hxr6k\" (UID: \"e6553d8c-9a9e-40be-a703-77d9c5235122\") " pod="openshift-cluster-node-tuning-operator/tuned-hxr6k"
Jul 23 11:35:47.149743 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:47.147711    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/7a2f8e1f-040a-4e01-88a7-9909e2fbc8b3-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-prdbw\" (UID: \"7a2f8e1f-040a-4e01-88a7-9909e2fbc8b3\") " pod="openshift-multus/multus-additional-cni-plugins-prdbw"
Jul 23 11:35:47.149743 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:47.148054    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0075cfcb-3377-4888-b93b-1fe1e8f4b6da-cni-binary-copy\") pod \"multus-rknqz\" (UID: \"0075cfcb-3377-4888-b93b-1fe1e8f4b6da\") " pod="openshift-multus/multus-rknqz"
Jul 23 11:35:47.149743 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:47.148240    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/7a2f8e1f-040a-4e01-88a7-9909e2fbc8b3-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-prdbw\" (UID: \"7a2f8e1f-040a-4e01-88a7-9909e2fbc8b3\") " pod="openshift-multus/multus-additional-cni-plugins-prdbw"
Jul 23 11:35:47.149743 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:47.148983    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/73f77fae-fdff-4232-a26a-e5749f7df186-agent-certs\") pod \"konnectivity-agent-2rs9k\" (UID: \"73f77fae-fdff-4232-a26a-e5749f7df186\") " pod="kube-system/konnectivity-agent-2rs9k"
Jul 23 11:35:47.149743 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:47.149038    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e6553d8c-9a9e-40be-a703-77d9c5235122-tmp\") pod \"tuned-hxr6k\" (UID: \"e6553d8c-9a9e-40be-a703-77d9c5235122\") " pod="openshift-cluster-node-tuning-operator/tuned-hxr6k"
Jul 23 11:35:47.149743 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:47.149601    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e6553d8c-9a9e-40be-a703-77d9c5235122-etc-tuned\") pod \"tuned-hxr6k\" (UID: \"e6553d8c-9a9e-40be-a703-77d9c5235122\") " pod="openshift-cluster-node-tuning-operator/tuned-hxr6k"
Jul 23 11:35:47.157630 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:47.157598    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zxvw\" (UniqueName: \"kubernetes.io/projected/7a2f8e1f-040a-4e01-88a7-9909e2fbc8b3-kube-api-access-4zxvw\") pod \"multus-additional-cni-plugins-prdbw\" (UID: \"7a2f8e1f-040a-4e01-88a7-9909e2fbc8b3\") " pod="openshift-multus/multus-additional-cni-plugins-prdbw"
Jul 23 11:35:47.158061 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:47.158033    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xn75t\" (UniqueName: \"kubernetes.io/projected/2ba42487-6ccf-4e15-93df-65f0623c41ba-kube-api-access-xn75t\") pod \"node-ca-dg4zg\" (UID: \"2ba42487-6ccf-4e15-93df-65f0623c41ba\") " pod="openshift-image-registry/node-ca-dg4zg"
Jul 23 11:35:47.158482 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:47.158442    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-blmdc\" (UniqueName: \"kubernetes.io/projected/e6553d8c-9a9e-40be-a703-77d9c5235122-kube-api-access-blmdc\") pod \"tuned-hxr6k\" (UID: \"e6553d8c-9a9e-40be-a703-77d9c5235122\") " pod="openshift-cluster-node-tuning-operator/tuned-hxr6k"
Jul 23 11:35:47.158823 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:47.158800    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rn28c\" (UniqueName: \"kubernetes.io/projected/cbdff1cb-47fd-4ed9-bec9-419ea8da3aaa-kube-api-access-rn28c\") pod \"network-metrics-daemon-pbmtq\" (UID: \"cbdff1cb-47fd-4ed9-bec9-419ea8da3aaa\") " pod="openshift-multus/network-metrics-daemon-pbmtq"
Jul 23 11:35:47.160052 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:47.160030    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxpkb\" (UniqueName: \"kubernetes.io/projected/0075cfcb-3377-4888-b93b-1fe1e8f4b6da-kube-api-access-qxpkb\") pod \"multus-rknqz\" (UID: \"0075cfcb-3377-4888-b93b-1fe1e8f4b6da\") " pod="openshift-multus/multus-rknqz"
Jul 23 11:35:47.161122 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:47.161094    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s75l6\" (UniqueName: \"kubernetes.io/projected/6bf989cb-53d0-4895-8bd1-377fe1451b7b-kube-api-access-s75l6\") pod \"aws-ebs-csi-driver-node-ljhws\" (UID: \"6bf989cb-53d0-4895-8bd1-377fe1451b7b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ljhws"
Jul 23 11:35:47.161229 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:47.161210    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8kc6s\" (UniqueName: \"kubernetes.io/projected/5ef3b70f-a117-4dcd-9380-5c5d36e596c0-kube-api-access-8kc6s\") pod \"node-resolver-dmnft\" (UID: \"5ef3b70f-a117-4dcd-9380-5c5d36e596c0\") " pod="openshift-dns/node-resolver-dmnft"
Jul 23 11:35:47.272091 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:47.271987    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-cb7gq"
Jul 23 11:35:47.281554 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:47.281519    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0bcf7e84_67a4_4cf6_bb1e_016bb38f4ccb.slice/crio-e027ea729cba1354d61066076433960031650ccf22dfd481293863d6aab39d3f WatchSource:0}: Error finding container e027ea729cba1354d61066076433960031650ccf22dfd481293863d6aab39d3f: Status 404 returned error can't find the container with id e027ea729cba1354d61066076433960031650ccf22dfd481293863d6aab39d3f
Jul 23 11:35:47.289748 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:47.289723    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-nhm9p"
Jul 23 11:35:47.296901 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:47.296880    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93649ac9_9062_46c0_adaa_1048d471211a.slice/crio-932593a5c84230ac070b5db01588dd83adc1c8467ac1f1f9f026387ab66eab00 WatchSource:0}: Error finding container 932593a5c84230ac070b5db01588dd83adc1c8467ac1f1f9f026387ab66eab00: Status 404 returned error can't find the container with id 932593a5c84230ac070b5db01588dd83adc1c8467ac1f1f9f026387ab66eab00
Jul 23 11:35:47.302393 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:47.302371    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-2rs9k"
Jul 23 11:35:47.309601 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:47.309572    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73f77fae_fdff_4232_a26a_e5749f7df186.slice/crio-6a1c72b592e498a95445bd3caea72b5a04dde6668cef62200a2ca18d459513aa WatchSource:0}: Error finding container 6a1c72b592e498a95445bd3caea72b5a04dde6668cef62200a2ca18d459513aa: Status 404 returned error can't find the container with id 6a1c72b592e498a95445bd3caea72b5a04dde6668cef62200a2ca18d459513aa
Jul 23 11:35:47.312572 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:47.312548    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ljhws"
Jul 23 11:35:47.320056 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:47.320035    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bf989cb_53d0_4895_8bd1_377fe1451b7b.slice/crio-f9d17896a6eac1be04b5d46591c0c9aeb2b08e904eccd10cbe3501b3c960afdb WatchSource:0}: Error finding container f9d17896a6eac1be04b5d46591c0c9aeb2b08e904eccd10cbe3501b3c960afdb: Status 404 returned error can't find the container with id f9d17896a6eac1be04b5d46591c0c9aeb2b08e904eccd10cbe3501b3c960afdb
Jul 23 11:35:47.328645 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:47.328615    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-hxr6k"
Jul 23 11:35:47.336601 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:47.336576    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6553d8c_9a9e_40be_a703_77d9c5235122.slice/crio-e27c39145ab46d895d943a9944bc118b6880e3ea06ce2885be71e8c1514adf81 WatchSource:0}: Error finding container e27c39145ab46d895d943a9944bc118b6880e3ea06ce2885be71e8c1514adf81: Status 404 returned error can't find the container with id e27c39145ab46d895d943a9944bc118b6880e3ea06ce2885be71e8c1514adf81
Jul 23 11:35:47.339849 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:47.339832    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-dmnft"
Jul 23 11:35:47.348035 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:47.348009    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ef3b70f_a117_4dcd_9380_5c5d36e596c0.slice/crio-62a5255a4535e706992b2e7a1644592c7bebc590d91990ea555d263711c5ea7b WatchSource:0}: Error finding container 62a5255a4535e706992b2e7a1644592c7bebc590d91990ea555d263711c5ea7b: Status 404 returned error can't find the container with id 62a5255a4535e706992b2e7a1644592c7bebc590d91990ea555d263711c5ea7b
Jul 23 11:35:47.358639 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:47.358621    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-dg4zg"
Jul 23 11:35:47.364815 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:47.364791    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-prdbw"
Jul 23 11:35:47.366545 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:47.366524    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ba42487_6ccf_4e15_93df_65f0623c41ba.slice/crio-47c7eb281c052979a3dad652d39996ea27052e016bd1861a65738952ca9bcb23 WatchSource:0}: Error finding container 47c7eb281c052979a3dad652d39996ea27052e016bd1861a65738952ca9bcb23: Status 404 returned error can't find the container with id 47c7eb281c052979a3dad652d39996ea27052e016bd1861a65738952ca9bcb23
Jul 23 11:35:47.369791 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:47.369767    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-rknqz"
Jul 23 11:35:47.378988 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:35:47.378967    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0075cfcb_3377_4888_b93b_1fe1e8f4b6da.slice/crio-dc4e068734cd91531b10d05fb8f6fbe136a7b8a287fcde1622e5c1e28e218179 WatchSource:0}: Error finding container dc4e068734cd91531b10d05fb8f6fbe136a7b8a287fcde1622e5c1e28e218179: Status 404 returned error can't find the container with id dc4e068734cd91531b10d05fb8f6fbe136a7b8a287fcde1622e5c1e28e218179
Jul 23 11:35:47.390589 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:47.390568    2563 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 23 11:35:47.651552 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:47.651521    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7a0946cf-0f04-49d5-ab74-6f64a48f8e5d-original-pull-secret\") pod \"global-pull-secret-syncer-cvbbt\" (UID: \"7a0946cf-0f04-49d5-ab74-6f64a48f8e5d\") " pod="kube-system/global-pull-secret-syncer-cvbbt"
Jul 23 11:35:47.651762 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:47.651582    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-528jd\" (UniqueName: \"kubernetes.io/projected/b30ea3bc-3059-4d68-934c-32cde3d06fbb-kube-api-access-528jd\") pod \"network-check-target-l6jwv\" (UID: \"b30ea3bc-3059-4d68-934c-32cde3d06fbb\") " pod="openshift-network-diagnostics/network-check-target-l6jwv"
Jul 23 11:35:47.651762 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:47.651619    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cbdff1cb-47fd-4ed9-bec9-419ea8da3aaa-metrics-certs\") pod \"network-metrics-daemon-pbmtq\" (UID: \"cbdff1cb-47fd-4ed9-bec9-419ea8da3aaa\") " pod="openshift-multus/network-metrics-daemon-pbmtq"
Jul 23 11:35:47.651910 ip-10-0-141-229 kubenswrapper[2563]: E0723 11:35:47.651772    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 23 11:35:47.651910 ip-10-0-141-229 kubenswrapper[2563]: E0723 11:35:47.651688    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 23 11:35:47.651910 ip-10-0-141-229 kubenswrapper[2563]: E0723 11:35:47.651799    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 23 11:35:47.651910 ip-10-0-141-229 kubenswrapper[2563]: E0723 11:35:47.651823    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 23 11:35:47.651910 ip-10-0-141-229 kubenswrapper[2563]: E0723 11:35:47.651837    2563 projected.go:196] Error preparing data for projected volume kube-api-access-528jd for pod openshift-network-diagnostics/network-check-target-l6jwv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 23 11:35:47.651910 ip-10-0-141-229 kubenswrapper[2563]: E0723 11:35:47.651844    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cbdff1cb-47fd-4ed9-bec9-419ea8da3aaa-metrics-certs podName:cbdff1cb-47fd-4ed9-bec9-419ea8da3aaa nodeName:}" failed. No retries permitted until 2026-07-23 11:35:48.65182357 +0000 UTC m=+3.044761357 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cbdff1cb-47fd-4ed9-bec9-419ea8da3aaa-metrics-certs") pod "network-metrics-daemon-pbmtq" (UID: "cbdff1cb-47fd-4ed9-bec9-419ea8da3aaa") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 23 11:35:47.651910 ip-10-0-141-229 kubenswrapper[2563]: E0723 11:35:47.651864    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7a0946cf-0f04-49d5-ab74-6f64a48f8e5d-original-pull-secret podName:7a0946cf-0f04-49d5-ab74-6f64a48f8e5d nodeName:}" failed. No retries permitted until 2026-07-23 11:35:48.651853721 +0000 UTC m=+3.044791490 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7a0946cf-0f04-49d5-ab74-6f64a48f8e5d-original-pull-secret") pod "global-pull-secret-syncer-cvbbt" (UID: "7a0946cf-0f04-49d5-ab74-6f64a48f8e5d") : object "kube-system"/"original-pull-secret" not registered
Jul 23 11:35:47.651910 ip-10-0-141-229 kubenswrapper[2563]: E0723 11:35:47.651881    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b30ea3bc-3059-4d68-934c-32cde3d06fbb-kube-api-access-528jd podName:b30ea3bc-3059-4d68-934c-32cde3d06fbb nodeName:}" failed. No retries permitted until 2026-07-23 11:35:48.651870972 +0000 UTC m=+3.044808743 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-528jd" (UniqueName: "kubernetes.io/projected/b30ea3bc-3059-4d68-934c-32cde3d06fbb-kube-api-access-528jd") pod "network-check-target-l6jwv" (UID: "b30ea3bc-3059-4d68-934c-32cde3d06fbb") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 23 11:35:47.950722 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:47.950473    2563 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 23 11:35:47.987575 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:47.987399    2563 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-22 11:30:47 +0000 UTC" deadline="2028-04-09 21:58:48.452971752 +0000 UTC"
Jul 23 11:35:47.987575 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:47.987463    2563 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15034h23m0.465513375s"
Jul 23 11:35:48.072495 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:48.072324    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pbmtq"
Jul 23 11:35:48.072495 ip-10-0-141-229 kubenswrapper[2563]: E0723 11:35:48.072490    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pbmtq" podUID="cbdff1cb-47fd-4ed9-bec9-419ea8da3aaa"
Jul 23 11:35:48.090368 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:48.090279    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-dmnft" event={"ID":"5ef3b70f-a117-4dcd-9380-5c5d36e596c0","Type":"ContainerStarted","Data":"62a5255a4535e706992b2e7a1644592c7bebc590d91990ea555d263711c5ea7b"}
Jul 23 11:35:48.108034 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:48.107967    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ljhws" event={"ID":"6bf989cb-53d0-4895-8bd1-377fe1451b7b","Type":"ContainerStarted","Data":"f9d17896a6eac1be04b5d46591c0c9aeb2b08e904eccd10cbe3501b3c960afdb"}
Jul 23 11:35:48.113186 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:48.113151    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-cb7gq" event={"ID":"0bcf7e84-67a4-4cf6-bb1e-016bb38f4ccb","Type":"ContainerStarted","Data":"e027ea729cba1354d61066076433960031650ccf22dfd481293863d6aab39d3f"}
Jul 23 11:35:48.122705 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:48.122515    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-dg4zg" event={"ID":"2ba42487-6ccf-4e15-93df-65f0623c41ba","Type":"ContainerStarted","Data":"47c7eb281c052979a3dad652d39996ea27052e016bd1861a65738952ca9bcb23"}
Jul 23 11:35:48.131058 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:48.131025    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-hxr6k" event={"ID":"e6553d8c-9a9e-40be-a703-77d9c5235122","Type":"ContainerStarted","Data":"e27c39145ab46d895d943a9944bc118b6880e3ea06ce2885be71e8c1514adf81"}
Jul 23 11:35:48.139013 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:48.138981    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-2rs9k" event={"ID":"73f77fae-fdff-4232-a26a-e5749f7df186","Type":"ContainerStarted","Data":"6a1c72b592e498a95445bd3caea72b5a04dde6668cef62200a2ca18d459513aa"}
Jul 23 11:35:48.153811 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:48.152191    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nhm9p" event={"ID":"93649ac9-9062-46c0-adaa-1048d471211a","Type":"ContainerStarted","Data":"932593a5c84230ac070b5db01588dd83adc1c8467ac1f1f9f026387ab66eab00"}
Jul 23 11:35:48.158650 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:48.158620    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-rknqz" event={"ID":"0075cfcb-3377-4888-b93b-1fe1e8f4b6da","Type":"ContainerStarted","Data":"dc4e068734cd91531b10d05fb8f6fbe136a7b8a287fcde1622e5c1e28e218179"}
Jul 23 11:35:48.181064 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:48.180974    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-prdbw" event={"ID":"7a2f8e1f-040a-4e01-88a7-9909e2fbc8b3","Type":"ContainerStarted","Data":"55524cc4b38f2327a032b54fd2d5a009a37020a68f08a3506a5bcd546943deb3"}
Jul 23 11:35:48.660981 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:48.660943    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-528jd\" (UniqueName: \"kubernetes.io/projected/b30ea3bc-3059-4d68-934c-32cde3d06fbb-kube-api-access-528jd\") pod \"network-check-target-l6jwv\" (UID: \"b30ea3bc-3059-4d68-934c-32cde3d06fbb\") " pod="openshift-network-diagnostics/network-check-target-l6jwv"
Jul 23 11:35:48.661153 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:48.661001    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cbdff1cb-47fd-4ed9-bec9-419ea8da3aaa-metrics-certs\") pod \"network-metrics-daemon-pbmtq\" (UID: \"cbdff1cb-47fd-4ed9-bec9-419ea8da3aaa\") " pod="openshift-multus/network-metrics-daemon-pbmtq"
Jul 23 11:35:48.661153 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:48.661058    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7a0946cf-0f04-49d5-ab74-6f64a48f8e5d-original-pull-secret\") pod \"global-pull-secret-syncer-cvbbt\" (UID: \"7a0946cf-0f04-49d5-ab74-6f64a48f8e5d\") " pod="kube-system/global-pull-secret-syncer-cvbbt"
Jul 23 11:35:48.661281 ip-10-0-141-229 kubenswrapper[2563]: E0723 11:35:48.661187    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 23 11:35:48.661281 ip-10-0-141-229 kubenswrapper[2563]: E0723 11:35:48.661256    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7a0946cf-0f04-49d5-ab74-6f64a48f8e5d-original-pull-secret podName:7a0946cf-0f04-49d5-ab74-6f64a48f8e5d nodeName:}" failed. No retries permitted until 2026-07-23 11:35:50.661228902 +0000 UTC m=+5.054166672 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7a0946cf-0f04-49d5-ab74-6f64a48f8e5d-original-pull-secret") pod "global-pull-secret-syncer-cvbbt" (UID: "7a0946cf-0f04-49d5-ab74-6f64a48f8e5d") : object "kube-system"/"original-pull-secret" not registered
Jul 23 11:35:48.661393 ip-10-0-141-229 kubenswrapper[2563]: E0723 11:35:48.661347    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 23 11:35:48.661393 ip-10-0-141-229 kubenswrapper[2563]: E0723 11:35:48.661364    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 23 11:35:48.661393 ip-10-0-141-229 kubenswrapper[2563]: E0723 11:35:48.661377    2563 projected.go:196] Error preparing data for projected volume kube-api-access-528jd for pod openshift-network-diagnostics/network-check-target-l6jwv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 23 11:35:48.661536 ip-10-0-141-229 kubenswrapper[2563]: E0723 11:35:48.661413    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b30ea3bc-3059-4d68-934c-32cde3d06fbb-kube-api-access-528jd podName:b30ea3bc-3059-4d68-934c-32cde3d06fbb nodeName:}" failed. No retries permitted until 2026-07-23 11:35:50.661401787 +0000 UTC m=+5.054339556 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-528jd" (UniqueName: "kubernetes.io/projected/b30ea3bc-3059-4d68-934c-32cde3d06fbb-kube-api-access-528jd") pod "network-check-target-l6jwv" (UID: "b30ea3bc-3059-4d68-934c-32cde3d06fbb") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 23 11:35:48.661536 ip-10-0-141-229 kubenswrapper[2563]: E0723 11:35:48.661490    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 23 11:35:48.661536 ip-10-0-141-229 kubenswrapper[2563]: E0723 11:35:48.661523    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cbdff1cb-47fd-4ed9-bec9-419ea8da3aaa-metrics-certs podName:cbdff1cb-47fd-4ed9-bec9-419ea8da3aaa nodeName:}" failed. No retries permitted until 2026-07-23 11:35:50.661512526 +0000 UTC m=+5.054450297 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cbdff1cb-47fd-4ed9-bec9-419ea8da3aaa-metrics-certs") pod "network-metrics-daemon-pbmtq" (UID: "cbdff1cb-47fd-4ed9-bec9-419ea8da3aaa") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 23 11:35:48.988487 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:48.988417    2563 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-22 11:30:47 +0000 UTC" deadline="2028-02-07 15:07:40.044168977 +0000 UTC"
Jul 23 11:35:48.988487 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:48.988457    2563 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13539h31m51.055717183s"
Jul 23 11:35:49.072257 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:49.072219    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l6jwv"
Jul 23 11:35:49.072438 ip-10-0-141-229 kubenswrapper[2563]: E0723 11:35:49.072332    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l6jwv" podUID="b30ea3bc-3059-4d68-934c-32cde3d06fbb"
Jul 23 11:35:49.072720 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:49.072701    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cvbbt"
Jul 23 11:35:49.072816 ip-10-0-141-229 kubenswrapper[2563]: E0723 11:35:49.072779    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cvbbt" podUID="7a0946cf-0f04-49d5-ab74-6f64a48f8e5d"
Jul 23 11:35:49.987240 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:49.987212    2563 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 23 11:35:50.074729 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:50.074310    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pbmtq"
Jul 23 11:35:50.074729 ip-10-0-141-229 kubenswrapper[2563]: E0723 11:35:50.074457    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pbmtq" podUID="cbdff1cb-47fd-4ed9-bec9-419ea8da3aaa"
Jul 23 11:35:50.679720 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:50.679655    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7a0946cf-0f04-49d5-ab74-6f64a48f8e5d-original-pull-secret\") pod \"global-pull-secret-syncer-cvbbt\" (UID: \"7a0946cf-0f04-49d5-ab74-6f64a48f8e5d\") " pod="kube-system/global-pull-secret-syncer-cvbbt"
Jul 23 11:35:50.679912 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:50.679742    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-528jd\" (UniqueName: \"kubernetes.io/projected/b30ea3bc-3059-4d68-934c-32cde3d06fbb-kube-api-access-528jd\") pod \"network-check-target-l6jwv\" (UID: \"b30ea3bc-3059-4d68-934c-32cde3d06fbb\") " pod="openshift-network-diagnostics/network-check-target-l6jwv"
Jul 23 11:35:50.679912 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:50.679781    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cbdff1cb-47fd-4ed9-bec9-419ea8da3aaa-metrics-certs\") pod \"network-metrics-daemon-pbmtq\" (UID: \"cbdff1cb-47fd-4ed9-bec9-419ea8da3aaa\") " pod="openshift-multus/network-metrics-daemon-pbmtq"
Jul 23 11:35:50.680026 ip-10-0-141-229 kubenswrapper[2563]: E0723 11:35:50.679924    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 23 11:35:50.680026 ip-10-0-141-229 kubenswrapper[2563]: E0723 11:35:50.679989    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cbdff1cb-47fd-4ed9-bec9-419ea8da3aaa-metrics-certs podName:cbdff1cb-47fd-4ed9-bec9-419ea8da3aaa nodeName:}" failed. No retries permitted until 2026-07-23 11:35:54.679969034 +0000 UTC m=+9.072906819 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cbdff1cb-47fd-4ed9-bec9-419ea8da3aaa-metrics-certs") pod "network-metrics-daemon-pbmtq" (UID: "cbdff1cb-47fd-4ed9-bec9-419ea8da3aaa") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 23 11:35:50.680406 ip-10-0-141-229 kubenswrapper[2563]: E0723 11:35:50.680384    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 23 11:35:50.680508 ip-10-0-141-229 kubenswrapper[2563]: E0723 11:35:50.680441    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7a0946cf-0f04-49d5-ab74-6f64a48f8e5d-original-pull-secret podName:7a0946cf-0f04-49d5-ab74-6f64a48f8e5d nodeName:}" failed. No retries permitted until 2026-07-23 11:35:54.680425945 +0000 UTC m=+9.073363726 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7a0946cf-0f04-49d5-ab74-6f64a48f8e5d-original-pull-secret") pod "global-pull-secret-syncer-cvbbt" (UID: "7a0946cf-0f04-49d5-ab74-6f64a48f8e5d") : object "kube-system"/"original-pull-secret" not registered
Jul 23 11:35:50.680573 ip-10-0-141-229 kubenswrapper[2563]: E0723 11:35:50.680520    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 23 11:35:50.680573 ip-10-0-141-229 kubenswrapper[2563]: E0723 11:35:50.680534    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 23 11:35:50.680573 ip-10-0-141-229 kubenswrapper[2563]: E0723 11:35:50.680546    2563 projected.go:196] Error preparing data for projected volume kube-api-access-528jd for pod openshift-network-diagnostics/network-check-target-l6jwv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 23 11:35:50.680732 ip-10-0-141-229 kubenswrapper[2563]: E0723 11:35:50.680579    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b30ea3bc-3059-4d68-934c-32cde3d06fbb-kube-api-access-528jd podName:b30ea3bc-3059-4d68-934c-32cde3d06fbb nodeName:}" failed. No retries permitted until 2026-07-23 11:35:54.680569195 +0000 UTC m=+9.073506966 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-528jd" (UniqueName: "kubernetes.io/projected/b30ea3bc-3059-4d68-934c-32cde3d06fbb-kube-api-access-528jd") pod "network-check-target-l6jwv" (UID: "b30ea3bc-3059-4d68-934c-32cde3d06fbb") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 23 11:35:51.072274 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:51.072181    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l6jwv"
Jul 23 11:35:51.072435 ip-10-0-141-229 kubenswrapper[2563]: E0723 11:35:51.072311    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l6jwv" podUID="b30ea3bc-3059-4d68-934c-32cde3d06fbb"
Jul 23 11:35:51.072723 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:51.072702    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cvbbt"
Jul 23 11:35:51.072831 ip-10-0-141-229 kubenswrapper[2563]: E0723 11:35:51.072798    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cvbbt" podUID="7a0946cf-0f04-49d5-ab74-6f64a48f8e5d"
Jul 23 11:35:52.072188 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:52.072156    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pbmtq"
Jul 23 11:35:52.072650 ip-10-0-141-229 kubenswrapper[2563]: E0723 11:35:52.072281    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pbmtq" podUID="cbdff1cb-47fd-4ed9-bec9-419ea8da3aaa"
Jul 23 11:35:53.071891 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:53.071839    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cvbbt"
Jul 23 11:35:53.072072 ip-10-0-141-229 kubenswrapper[2563]: E0723 11:35:53.071981    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cvbbt" podUID="7a0946cf-0f04-49d5-ab74-6f64a48f8e5d"
Jul 23 11:35:53.072484 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:53.072464    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l6jwv"
Jul 23 11:35:53.072876 ip-10-0-141-229 kubenswrapper[2563]: E0723 11:35:53.072604    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l6jwv" podUID="b30ea3bc-3059-4d68-934c-32cde3d06fbb"
Jul 23 11:35:54.072744 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:54.072711    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pbmtq"
Jul 23 11:35:54.073191 ip-10-0-141-229 kubenswrapper[2563]: E0723 11:35:54.072853    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pbmtq" podUID="cbdff1cb-47fd-4ed9-bec9-419ea8da3aaa"
Jul 23 11:35:54.716878 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:54.716839    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7a0946cf-0f04-49d5-ab74-6f64a48f8e5d-original-pull-secret\") pod \"global-pull-secret-syncer-cvbbt\" (UID: \"7a0946cf-0f04-49d5-ab74-6f64a48f8e5d\") " pod="kube-system/global-pull-secret-syncer-cvbbt"
Jul 23 11:35:54.717048 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:54.716906    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-528jd\" (UniqueName: \"kubernetes.io/projected/b30ea3bc-3059-4d68-934c-32cde3d06fbb-kube-api-access-528jd\") pod \"network-check-target-l6jwv\" (UID: \"b30ea3bc-3059-4d68-934c-32cde3d06fbb\") " pod="openshift-network-diagnostics/network-check-target-l6jwv"
Jul 23 11:35:54.717048 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:54.716945    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cbdff1cb-47fd-4ed9-bec9-419ea8da3aaa-metrics-certs\") pod \"network-metrics-daemon-pbmtq\" (UID: \"cbdff1cb-47fd-4ed9-bec9-419ea8da3aaa\") " pod="openshift-multus/network-metrics-daemon-pbmtq"
Jul 23 11:35:54.717173 ip-10-0-141-229 kubenswrapper[2563]: E0723 11:35:54.717054    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 23 11:35:54.717173 ip-10-0-141-229 kubenswrapper[2563]: E0723 11:35:54.717108    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cbdff1cb-47fd-4ed9-bec9-419ea8da3aaa-metrics-certs podName:cbdff1cb-47fd-4ed9-bec9-419ea8da3aaa nodeName:}" failed. No retries permitted until 2026-07-23 11:36:02.717094335 +0000 UTC m=+17.110032101 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cbdff1cb-47fd-4ed9-bec9-419ea8da3aaa-metrics-certs") pod "network-metrics-daemon-pbmtq" (UID: "cbdff1cb-47fd-4ed9-bec9-419ea8da3aaa") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 23 11:35:54.717602 ip-10-0-141-229 kubenswrapper[2563]: E0723 11:35:54.717432    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 23 11:35:54.717602 ip-10-0-141-229 kubenswrapper[2563]: E0723 11:35:54.717485    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7a0946cf-0f04-49d5-ab74-6f64a48f8e5d-original-pull-secret podName:7a0946cf-0f04-49d5-ab74-6f64a48f8e5d nodeName:}" failed. No retries permitted until 2026-07-23 11:36:02.717469761 +0000 UTC m=+17.110407543 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7a0946cf-0f04-49d5-ab74-6f64a48f8e5d-original-pull-secret") pod "global-pull-secret-syncer-cvbbt" (UID: "7a0946cf-0f04-49d5-ab74-6f64a48f8e5d") : object "kube-system"/"original-pull-secret" not registered
Jul 23 11:35:54.726628 ip-10-0-141-229 kubenswrapper[2563]: E0723 11:35:54.718169    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 23 11:35:54.726628 ip-10-0-141-229 kubenswrapper[2563]: E0723 11:35:54.718198    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 23 11:35:54.726628 ip-10-0-141-229 kubenswrapper[2563]: E0723 11:35:54.718212    2563 projected.go:196] Error preparing data for projected volume kube-api-access-528jd for pod openshift-network-diagnostics/network-check-target-l6jwv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 23 11:35:54.726628 ip-10-0-141-229 kubenswrapper[2563]: E0723 11:35:54.718294    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b30ea3bc-3059-4d68-934c-32cde3d06fbb-kube-api-access-528jd podName:b30ea3bc-3059-4d68-934c-32cde3d06fbb nodeName:}" failed. No retries permitted until 2026-07-23 11:36:02.718275454 +0000 UTC m=+17.111213286 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-528jd" (UniqueName: "kubernetes.io/projected/b30ea3bc-3059-4d68-934c-32cde3d06fbb-kube-api-access-528jd") pod "network-check-target-l6jwv" (UID: "b30ea3bc-3059-4d68-934c-32cde3d06fbb") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 23 11:35:55.072081 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:55.071973    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cvbbt"
Jul 23 11:35:55.072081 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:55.072001    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l6jwv"
Jul 23 11:35:55.072270 ip-10-0-141-229 kubenswrapper[2563]: E0723 11:35:55.072104    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cvbbt" podUID="7a0946cf-0f04-49d5-ab74-6f64a48f8e5d"
Jul 23 11:35:55.072482 ip-10-0-141-229 kubenswrapper[2563]: E0723 11:35:55.072437    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l6jwv" podUID="b30ea3bc-3059-4d68-934c-32cde3d06fbb"
Jul 23 11:35:56.073162 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:56.072698    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pbmtq"
Jul 23 11:35:56.073162 ip-10-0-141-229 kubenswrapper[2563]: E0723 11:35:56.072816    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pbmtq" podUID="cbdff1cb-47fd-4ed9-bec9-419ea8da3aaa"
Jul 23 11:35:57.072836 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:57.072194    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cvbbt"
Jul 23 11:35:57.072836 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:57.072243    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l6jwv"
Jul 23 11:35:57.072836 ip-10-0-141-229 kubenswrapper[2563]: E0723 11:35:57.072338    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cvbbt" podUID="7a0946cf-0f04-49d5-ab74-6f64a48f8e5d"
Jul 23 11:35:57.072836 ip-10-0-141-229 kubenswrapper[2563]: E0723 11:35:57.072784    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l6jwv" podUID="b30ea3bc-3059-4d68-934c-32cde3d06fbb"
Jul 23 11:35:58.072556 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:58.072522    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pbmtq"
Jul 23 11:35:58.072972 ip-10-0-141-229 kubenswrapper[2563]: E0723 11:35:58.072663    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pbmtq" podUID="cbdff1cb-47fd-4ed9-bec9-419ea8da3aaa"
Jul 23 11:35:59.072443 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:59.072403    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l6jwv"
Jul 23 11:35:59.072599 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:35:59.072419    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cvbbt"
Jul 23 11:35:59.072599 ip-10-0-141-229 kubenswrapper[2563]: E0723 11:35:59.072506    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l6jwv" podUID="b30ea3bc-3059-4d68-934c-32cde3d06fbb"
Jul 23 11:35:59.072920 ip-10-0-141-229 kubenswrapper[2563]: E0723 11:35:59.072608    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cvbbt" podUID="7a0946cf-0f04-49d5-ab74-6f64a48f8e5d"
Jul 23 11:36:00.072134 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:00.072095    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pbmtq"
Jul 23 11:36:00.072321 ip-10-0-141-229 kubenswrapper[2563]: E0723 11:36:00.072246    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pbmtq" podUID="cbdff1cb-47fd-4ed9-bec9-419ea8da3aaa"
Jul 23 11:36:01.072143 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:01.072111    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l6jwv"
Jul 23 11:36:01.072592 ip-10-0-141-229 kubenswrapper[2563]: E0723 11:36:01.072219    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l6jwv" podUID="b30ea3bc-3059-4d68-934c-32cde3d06fbb"
Jul 23 11:36:01.072592 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:01.072112    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cvbbt"
Jul 23 11:36:01.072677 ip-10-0-141-229 kubenswrapper[2563]: E0723 11:36:01.072596    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cvbbt" podUID="7a0946cf-0f04-49d5-ab74-6f64a48f8e5d"
Jul 23 11:36:02.071905 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:02.071870    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pbmtq"
Jul 23 11:36:02.072094 ip-10-0-141-229 kubenswrapper[2563]: E0723 11:36:02.072011    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pbmtq" podUID="cbdff1cb-47fd-4ed9-bec9-419ea8da3aaa"
Jul 23 11:36:02.778504 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:02.778467    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7a0946cf-0f04-49d5-ab74-6f64a48f8e5d-original-pull-secret\") pod \"global-pull-secret-syncer-cvbbt\" (UID: \"7a0946cf-0f04-49d5-ab74-6f64a48f8e5d\") " pod="kube-system/global-pull-secret-syncer-cvbbt"
Jul 23 11:36:02.779005 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:02.778531    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-528jd\" (UniqueName: \"kubernetes.io/projected/b30ea3bc-3059-4d68-934c-32cde3d06fbb-kube-api-access-528jd\") pod \"network-check-target-l6jwv\" (UID: \"b30ea3bc-3059-4d68-934c-32cde3d06fbb\") " pod="openshift-network-diagnostics/network-check-target-l6jwv"
Jul 23 11:36:02.779005 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:02.778565    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cbdff1cb-47fd-4ed9-bec9-419ea8da3aaa-metrics-certs\") pod \"network-metrics-daemon-pbmtq\" (UID: \"cbdff1cb-47fd-4ed9-bec9-419ea8da3aaa\") " pod="openshift-multus/network-metrics-daemon-pbmtq"
Jul 23 11:36:02.779005 ip-10-0-141-229 kubenswrapper[2563]: E0723 11:36:02.778630    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 23 11:36:02.779005 ip-10-0-141-229 kubenswrapper[2563]: E0723 11:36:02.778714    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 23 11:36:02.779005 ip-10-0-141-229 kubenswrapper[2563]: E0723 11:36:02.778830    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 23 11:36:02.779005 ip-10-0-141-229 kubenswrapper[2563]: E0723 11:36:02.778846    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 23 11:36:02.779005 ip-10-0-141-229 kubenswrapper[2563]: E0723 11:36:02.778860    2563 projected.go:196] Error preparing data for projected volume kube-api-access-528jd for pod openshift-network-diagnostics/network-check-target-l6jwv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 23 11:36:02.779005 ip-10-0-141-229 kubenswrapper[2563]: E0723 11:36:02.778715    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7a0946cf-0f04-49d5-ab74-6f64a48f8e5d-original-pull-secret podName:7a0946cf-0f04-49d5-ab74-6f64a48f8e5d nodeName:}" failed. No retries permitted until 2026-07-23 11:36:18.778699356 +0000 UTC m=+33.171637135 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7a0946cf-0f04-49d5-ab74-6f64a48f8e5d-original-pull-secret") pod "global-pull-secret-syncer-cvbbt" (UID: "7a0946cf-0f04-49d5-ab74-6f64a48f8e5d") : object "kube-system"/"original-pull-secret" not registered
Jul 23 11:36:02.779005 ip-10-0-141-229 kubenswrapper[2563]: E0723 11:36:02.778911    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cbdff1cb-47fd-4ed9-bec9-419ea8da3aaa-metrics-certs podName:cbdff1cb-47fd-4ed9-bec9-419ea8da3aaa nodeName:}" failed. No retries permitted until 2026-07-23 11:36:18.778895306 +0000 UTC m=+33.171833073 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cbdff1cb-47fd-4ed9-bec9-419ea8da3aaa-metrics-certs") pod "network-metrics-daemon-pbmtq" (UID: "cbdff1cb-47fd-4ed9-bec9-419ea8da3aaa") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 23 11:36:02.779005 ip-10-0-141-229 kubenswrapper[2563]: E0723 11:36:02.778925    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b30ea3bc-3059-4d68-934c-32cde3d06fbb-kube-api-access-528jd podName:b30ea3bc-3059-4d68-934c-32cde3d06fbb nodeName:}" failed. No retries permitted until 2026-07-23 11:36:18.778916942 +0000 UTC m=+33.171854707 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-528jd" (UniqueName: "kubernetes.io/projected/b30ea3bc-3059-4d68-934c-32cde3d06fbb-kube-api-access-528jd") pod "network-check-target-l6jwv" (UID: "b30ea3bc-3059-4d68-934c-32cde3d06fbb") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 23 11:36:03.071904 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:03.071821    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cvbbt"
Jul 23 11:36:03.071904 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:03.071856    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l6jwv"
Jul 23 11:36:03.072114 ip-10-0-141-229 kubenswrapper[2563]: E0723 11:36:03.071966    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cvbbt" podUID="7a0946cf-0f04-49d5-ab74-6f64a48f8e5d"
Jul 23 11:36:03.072168 ip-10-0-141-229 kubenswrapper[2563]: E0723 11:36:03.072104    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l6jwv" podUID="b30ea3bc-3059-4d68-934c-32cde3d06fbb"
Jul 23 11:36:04.071932 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:04.071898    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pbmtq"
Jul 23 11:36:04.072414 ip-10-0-141-229 kubenswrapper[2563]: E0723 11:36:04.072048    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pbmtq" podUID="cbdff1cb-47fd-4ed9-bec9-419ea8da3aaa"
Jul 23 11:36:05.072139 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:05.072098    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l6jwv"
Jul 23 11:36:05.072139 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:05.072138    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cvbbt"
Jul 23 11:36:05.072607 ip-10-0-141-229 kubenswrapper[2563]: E0723 11:36:05.072224    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l6jwv" podUID="b30ea3bc-3059-4d68-934c-32cde3d06fbb"
Jul 23 11:36:05.072607 ip-10-0-141-229 kubenswrapper[2563]: E0723 11:36:05.072386    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cvbbt" podUID="7a0946cf-0f04-49d5-ab74-6f64a48f8e5d"
Jul 23 11:36:06.073378 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:06.073341    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pbmtq"
Jul 23 11:36:06.073849 ip-10-0-141-229 kubenswrapper[2563]: E0723 11:36:06.073456    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pbmtq" podUID="cbdff1cb-47fd-4ed9-bec9-419ea8da3aaa"
Jul 23 11:36:07.072168 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:07.071643    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cvbbt"
Jul 23 11:36:07.072168 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:07.071708    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l6jwv"
Jul 23 11:36:07.072168 ip-10-0-141-229 kubenswrapper[2563]: E0723 11:36:07.071802    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cvbbt" podUID="7a0946cf-0f04-49d5-ab74-6f64a48f8e5d"
Jul 23 11:36:07.072168 ip-10-0-141-229 kubenswrapper[2563]: E0723 11:36:07.071932    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l6jwv" podUID="b30ea3bc-3059-4d68-934c-32cde3d06fbb"
Jul 23 11:36:08.071899 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:08.071720    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pbmtq"
Jul 23 11:36:08.072290 ip-10-0-141-229 kubenswrapper[2563]: E0723 11:36:08.071983    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pbmtq" podUID="cbdff1cb-47fd-4ed9-bec9-419ea8da3aaa"
Jul 23 11:36:08.233937 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:08.233753    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nhm9p_93649ac9-9062-46c0-adaa-1048d471211a/ovn-acl-logging/0.log"
Jul 23 11:36:08.235181 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:08.235088    2563 generic.go:356] "Generic (PLEG): container finished" podID="93649ac9-9062-46c0-adaa-1048d471211a" containerID="09f5dcb10e85220fd9bcefe2c22efca306cc49dac02c68b5b9dcda34cee5a6e9" exitCode=1
Jul 23 11:36:08.235283 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:08.235166    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nhm9p" event={"ID":"93649ac9-9062-46c0-adaa-1048d471211a","Type":"ContainerStarted","Data":"98739e2fda6ddec883ccaa6f7ee3ab80fb72068948b6b697cc4cb29133eea6ec"}
Jul 23 11:36:08.235283 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:08.235263    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nhm9p" event={"ID":"93649ac9-9062-46c0-adaa-1048d471211a","Type":"ContainerStarted","Data":"d34a0e517e492e3b950ef8fb5d42c3ce2664351d991b5dcdbb1d7bd46be1d4da"}
Jul 23 11:36:08.235391 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:08.235325    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nhm9p" event={"ID":"93649ac9-9062-46c0-adaa-1048d471211a","Type":"ContainerDied","Data":"09f5dcb10e85220fd9bcefe2c22efca306cc49dac02c68b5b9dcda34cee5a6e9"}
Jul 23 11:36:08.235498 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:08.235481    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nhm9p" event={"ID":"93649ac9-9062-46c0-adaa-1048d471211a","Type":"ContainerStarted","Data":"7d9ce9d388bd2a1bcfcc5193d6d48cbbaeb040f01ed86bd19a9a0cf8938868be"}
Jul 23 11:36:08.237633 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:08.237206    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-229.ec2.internal" event={"ID":"192fe6abc0214ce0c06c38f944481f45","Type":"ContainerStarted","Data":"987a4bb98db7912499af275d14091f71329889caa91a4927c9f2a18df69ebde2"}
Jul 23 11:36:08.238911 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:08.238887    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-rknqz" event={"ID":"0075cfcb-3377-4888-b93b-1fe1e8f4b6da","Type":"ContainerStarted","Data":"3cde7d6762e5a736fdf8ab4290b9b0642bccd5d7d22555a3381d51c7050eef48"}
Jul 23 11:36:08.240609 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:08.240571    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-hxr6k" event={"ID":"e6553d8c-9a9e-40be-a703-77d9c5235122","Type":"ContainerStarted","Data":"b5bf8b6e5490bfec982506650cb6c7ad1495640dfa88070e68569145673ccd2a"}
Jul 23 11:36:08.253121 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:08.253062    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-229.ec2.internal" podStartSLOduration=22.253042727 podStartE2EDuration="22.253042727s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 11:35:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:36:08.252238152 +0000 UTC m=+22.645175939" watchObservedRunningTime="2026-07-23 11:36:08.253042727 +0000 UTC m=+22.645980516"
Jul 23 11:36:08.270669 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:08.270536    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-hxr6k" podStartSLOduration=1.938905187 podStartE2EDuration="22.270516636s" totalImagesPullingTime="20.331611449s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 11:35:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:36:08.270374709 +0000 UTC m=+22.663312497" watchObservedRunningTime="2026-07-23 11:36:08.270516636 +0000 UTC m=+22.663454423"
Jul 23 11:36:08.295260 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:08.295207    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-rknqz" podStartSLOduration=1.950927884 podStartE2EDuration="22.295145914s" totalImagesPullingTime="20.34421803s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 11:35:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:36:08.294633445 +0000 UTC m=+22.687571235" watchObservedRunningTime="2026-07-23 11:36:08.295145914 +0000 UTC m=+22.688083703"
Jul 23 11:36:09.072590 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:09.072557    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l6jwv"
Jul 23 11:36:09.073361 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:09.072557    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cvbbt"
Jul 23 11:36:09.073361 ip-10-0-141-229 kubenswrapper[2563]: E0723 11:36:09.072678    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l6jwv" podUID="b30ea3bc-3059-4d68-934c-32cde3d06fbb"
Jul 23 11:36:09.073361 ip-10-0-141-229 kubenswrapper[2563]: E0723 11:36:09.072809    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cvbbt" podUID="7a0946cf-0f04-49d5-ab74-6f64a48f8e5d"
Jul 23 11:36:09.244863 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:09.244770    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-cb7gq" event={"ID":"0bcf7e84-67a4-4cf6-bb1e-016bb38f4ccb","Type":"ContainerStarted","Data":"48f5dfc33a0b9b56a65bbcc2c7cf199a0eaf377b535ecf7a4a06755d30935f3a"}
Jul 23 11:36:09.246325 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:09.246291    2563 generic.go:356] "Generic (PLEG): container finished" podID="92a19fc888e8076d0b3fbd4b9dc96a0a" containerID="14899c21a6707ebf3fba005795bcca9711a92229e1fff7ebf75c66a258447baf" exitCode=0
Jul 23 11:36:09.246462 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:09.246372    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-229.ec2.internal" event={"ID":"92a19fc888e8076d0b3fbd4b9dc96a0a","Type":"ContainerDied","Data":"14899c21a6707ebf3fba005795bcca9711a92229e1fff7ebf75c66a258447baf"}
Jul 23 11:36:09.247779 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:09.247757    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-dg4zg" event={"ID":"2ba42487-6ccf-4e15-93df-65f0623c41ba","Type":"ContainerStarted","Data":"d2c2250cb0feea048413ce8d5f41c82eecbbaa3df705e5a80ec7c89c0da065f5"}
Jul 23 11:36:09.249207 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:09.249186    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-2rs9k" event={"ID":"73f77fae-fdff-4232-a26a-e5749f7df186","Type":"ContainerStarted","Data":"05013b0a9c60249ae556c3d8a6758a08fe8e4ed021526cb75e7a4fc927f04216"}
Jul 23 11:36:09.251639 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:09.251621    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nhm9p_93649ac9-9062-46c0-adaa-1048d471211a/ovn-acl-logging/0.log"
Jul 23 11:36:09.251998 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:09.251980    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nhm9p" event={"ID":"93649ac9-9062-46c0-adaa-1048d471211a","Type":"ContainerStarted","Data":"86fb0652b6ffd8f5dfe8766f43903fa84f395a6623a46ed1dc8fa313c81ef837"}
Jul 23 11:36:09.252059 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:09.252045    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nhm9p" event={"ID":"93649ac9-9062-46c0-adaa-1048d471211a","Type":"ContainerStarted","Data":"4773634e3ed407b894b0eca0cbfb7cace0107561db83a8a2591bd01147d6f127"}
Jul 23 11:36:09.253269 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:09.253251    2563 generic.go:356] "Generic (PLEG): container finished" podID="7a2f8e1f-040a-4e01-88a7-9909e2fbc8b3" containerID="570b9e7ae54291a4a9cfb4758ce0f8053b358ccc168237134ea27e58a0e75821" exitCode=0
Jul 23 11:36:09.253344 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:09.253314    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-prdbw" event={"ID":"7a2f8e1f-040a-4e01-88a7-9909e2fbc8b3","Type":"ContainerDied","Data":"570b9e7ae54291a4a9cfb4758ce0f8053b358ccc168237134ea27e58a0e75821"}
Jul 23 11:36:09.254662 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:09.254641    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-dmnft" event={"ID":"5ef3b70f-a117-4dcd-9380-5c5d36e596c0","Type":"ContainerStarted","Data":"2bfbf225dedf43ffba5ba5b4d881eea86d9961827244c1d21f2aff7392172fda"}
Jul 23 11:36:09.255998 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:09.255973    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ljhws" event={"ID":"6bf989cb-53d0-4895-8bd1-377fe1451b7b","Type":"ContainerStarted","Data":"79b7ca5861779ecbe6f5fc6b63cd2e2133dd0e61370da015f2f9c3c078697a21"}
Jul 23 11:36:09.275870 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:09.275819    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-cb7gq" podStartSLOduration=2.8905913 podStartE2EDuration="23.275802414s" totalImagesPullingTime="20.385211114s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 11:35:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:36:09.260439374 +0000 UTC m=+23.653377145" watchObservedRunningTime="2026-07-23 11:36:09.275802414 +0000 UTC m=+23.668740201"
Jul 23 11:36:09.276054 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:09.276035    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-2rs9k" podStartSLOduration=2.9185102609999998 podStartE2EDuration="23.276030629s" totalImagesPullingTime="20.357520368s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 11:35:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:36:09.275455188 +0000 UTC m=+23.668392977" watchObservedRunningTime="2026-07-23 11:36:09.276030629 +0000 UTC m=+23.668968416"
Jul 23 11:36:09.309707 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:09.309636    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-dmnft" podStartSLOduration=2.979301833 podStartE2EDuration="23.30961669s" totalImagesPullingTime="20.330314857s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 11:35:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:36:09.293925421 +0000 UTC m=+23.686863211" watchObservedRunningTime="2026-07-23 11:36:09.30961669 +0000 UTC m=+23.702554476"
Jul 23 11:36:09.309867 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:09.309773    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-dg4zg" podStartSLOduration=3.009494263 podStartE2EDuration="23.309766461s" totalImagesPullingTime="20.300272198s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 11:35:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:36:09.309353154 +0000 UTC m=+23.702290943" watchObservedRunningTime="2026-07-23 11:36:09.309766461 +0000 UTC m=+23.702704250"
Jul 23 11:36:09.411297 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:09.411259    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:36:09.484115 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:09.484078    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:36:09.614679 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:09.614647    2563 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 23 11:36:10.005273 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:10.005165    2563 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-23T11:36:09.614668255Z","UUID":"b06c8e30-1f22-4350-9e31-e604075b2dfd","Handler":null,"Name":"","Endpoint":""}
Jul 23 11:36:10.007033 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:10.007011    2563 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 23 11:36:10.007133 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:10.007063    2563 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 23 11:36:10.072808 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:10.072527    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pbmtq"
Jul 23 11:36:10.072808 ip-10-0-141-229 kubenswrapper[2563]: E0723 11:36:10.072667    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pbmtq" podUID="cbdff1cb-47fd-4ed9-bec9-419ea8da3aaa"
Jul 23 11:36:10.260384 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:10.260309    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ljhws" event={"ID":"6bf989cb-53d0-4895-8bd1-377fe1451b7b","Type":"ContainerStarted","Data":"f9a6a69dc842f9bbbabee765b65dfae0b7f18301315a79fe73cf3e5f2dc8f8ee"}
Jul 23 11:36:10.262207 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:10.262177    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-229.ec2.internal" event={"ID":"92a19fc888e8076d0b3fbd4b9dc96a0a","Type":"ContainerStarted","Data":"32cad94709e8e7272fe4d20f831e8881c69b7cdf359d837c2be29062029423e0"}
Jul 23 11:36:10.277853 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:10.277792    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-229.ec2.internal" podStartSLOduration=24.27777143 podStartE2EDuration="24.27777143s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 11:35:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:36:10.277717684 +0000 UTC m=+24.670655472" watchObservedRunningTime="2026-07-23 11:36:10.27777143 +0000 UTC m=+24.670709220"
Jul 23 11:36:11.072445 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:11.072353    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l6jwv"
Jul 23 11:36:11.072445 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:11.072400    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cvbbt"
Jul 23 11:36:11.072805 ip-10-0-141-229 kubenswrapper[2563]: E0723 11:36:11.072463    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l6jwv" podUID="b30ea3bc-3059-4d68-934c-32cde3d06fbb"
Jul 23 11:36:11.072805 ip-10-0-141-229 kubenswrapper[2563]: E0723 11:36:11.072579    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cvbbt" podUID="7a0946cf-0f04-49d5-ab74-6f64a48f8e5d"
Jul 23 11:36:11.266705 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:11.266651    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ljhws" event={"ID":"6bf989cb-53d0-4895-8bd1-377fe1451b7b","Type":"ContainerStarted","Data":"41a1705966a780cdf746eb3b594fa2a9ac65a55b9037a0b3097ce270cecb8b42"}
Jul 23 11:36:11.269925 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:11.269895    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nhm9p_93649ac9-9062-46c0-adaa-1048d471211a/ovn-acl-logging/0.log"
Jul 23 11:36:11.271018 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:11.270990    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nhm9p" event={"ID":"93649ac9-9062-46c0-adaa-1048d471211a","Type":"ContainerStarted","Data":"475626073a8dae9d097683c09cd29bee768166fe091189836fb4984dcceca241"}
Jul 23 11:36:11.286261 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:11.286215    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ljhws" podStartSLOduration=2.6973222 podStartE2EDuration="25.28620112s" totalImagesPullingTime="22.58887892s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 11:35:46 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:36:11.286128755 +0000 UTC m=+25.679066556" watchObservedRunningTime="2026-07-23 11:36:11.28620112 +0000 UTC m=+25.679138930"
Jul 23 11:36:11.568673 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:11.568635    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-2rs9k"
Jul 23 11:36:11.569262 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:11.569238    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-2rs9k"
Jul 23 11:36:11.834437 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:11.834354    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:36:11.933259 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:11.933217    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:36:12.034962 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:12.034849    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:36:12.072469 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:12.072429    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pbmtq"
Jul 23 11:36:12.072635 ip-10-0-141-229 kubenswrapper[2563]: E0723 11:36:12.072590    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pbmtq" podUID="cbdff1cb-47fd-4ed9-bec9-419ea8da3aaa"
Jul 23 11:36:13.071828 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:13.071753    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cvbbt"
Jul 23 11:36:13.072244 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:13.071757    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l6jwv"
Jul 23 11:36:13.072244 ip-10-0-141-229 kubenswrapper[2563]: E0723 11:36:13.071869    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cvbbt" podUID="7a0946cf-0f04-49d5-ab74-6f64a48f8e5d"
Jul 23 11:36:13.072244 ip-10-0-141-229 kubenswrapper[2563]: E0723 11:36:13.071934    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l6jwv" podUID="b30ea3bc-3059-4d68-934c-32cde3d06fbb"
Jul 23 11:36:13.151441 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:13.151408    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:36:13.252666 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:13.252625    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:36:13.962209 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:13.962028    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-2rs9k"
Jul 23 11:36:13.962338 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:13.962311    2563 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Jul 23 11:36:13.962702 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:13.962648    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-2rs9k"
Jul 23 11:36:14.028481 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:14.028400    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:36:14.071642 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:14.071616    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pbmtq"
Jul 23 11:36:14.071812 ip-10-0-141-229 kubenswrapper[2563]: E0723 11:36:14.071756    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pbmtq" podUID="cbdff1cb-47fd-4ed9-bec9-419ea8da3aaa"
Jul 23 11:36:14.280093 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:14.280018    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nhm9p_93649ac9-9062-46c0-adaa-1048d471211a/ovn-acl-logging/0.log"
Jul 23 11:36:14.280976 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:14.280800    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nhm9p" event={"ID":"93649ac9-9062-46c0-adaa-1048d471211a","Type":"ContainerStarted","Data":"32defb046486b9c77cc531025f66de3e9549f4b0dc61d3f0d94617cd7000923e"}
Jul 23 11:36:14.281733 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:14.281713    2563 scope.go:117] "RemoveContainer" containerID="09f5dcb10e85220fd9bcefe2c22efca306cc49dac02c68b5b9dcda34cee5a6e9"
Jul 23 11:36:14.284851 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:14.282558    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-nhm9p"
Jul 23 11:36:14.284851 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:14.282645    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-nhm9p"
Jul 23 11:36:14.284851 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:14.282730    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-nhm9p"
Jul 23 11:36:14.286527 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:14.286499    2563 generic.go:356] "Generic (PLEG): container finished" podID="7a2f8e1f-040a-4e01-88a7-9909e2fbc8b3" containerID="9bb33111616ca6112a6cc6b58f12302fc7e973d76e0f0a15ff4a556bca30863c" exitCode=0
Jul 23 11:36:14.286614 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:14.286589    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-prdbw" event={"ID":"7a2f8e1f-040a-4e01-88a7-9909e2fbc8b3","Type":"ContainerDied","Data":"9bb33111616ca6112a6cc6b58f12302fc7e973d76e0f0a15ff4a556bca30863c"}
Jul 23 11:36:14.300947 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:14.300925    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-nhm9p"
Jul 23 11:36:14.304102 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:14.304083    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-nhm9p"
Jul 23 11:36:14.823287 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:14.823244    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:36:14.922977 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:14.922939    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:36:14.998019 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:14.997978    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:36:15.072551 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:15.072519    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cvbbt"
Jul 23 11:36:15.072752 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:15.072524    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l6jwv"
Jul 23 11:36:15.072752 ip-10-0-141-229 kubenswrapper[2563]: E0723 11:36:15.072651    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cvbbt" podUID="7a0946cf-0f04-49d5-ab74-6f64a48f8e5d"
Jul 23 11:36:15.072752 ip-10-0-141-229 kubenswrapper[2563]: E0723 11:36:15.072725    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l6jwv" podUID="b30ea3bc-3059-4d68-934c-32cde3d06fbb"
Jul 23 11:36:15.292579 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:15.292554    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nhm9p_93649ac9-9062-46c0-adaa-1048d471211a/ovn-acl-logging/0.log"
Jul 23 11:36:15.293200 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:15.292889    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nhm9p" event={"ID":"93649ac9-9062-46c0-adaa-1048d471211a","Type":"ContainerStarted","Data":"c5f718cee76aed2633424b0ffa5008b5bd95cece8940cbf1e780f5d3f7d2538f"}
Jul 23 11:36:15.365594 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:15.363761    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-nhm9p" podStartSLOduration=8.919454722 podStartE2EDuration="29.363739126s" totalImagesPullingTime="20.444284404s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 11:35:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:36:15.360915019 +0000 UTC m=+29.753852800" watchObservedRunningTime="2026-07-23 11:36:15.363739126 +0000 UTC m=+29.756676915"
Jul 23 11:36:15.668047 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:15.668012    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-pbmtq"]
Jul 23 11:36:15.668299 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:15.668243    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pbmtq"
Jul 23 11:36:15.668412 ip-10-0-141-229 kubenswrapper[2563]: E0723 11:36:15.668386    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pbmtq" podUID="cbdff1cb-47fd-4ed9-bec9-419ea8da3aaa"
Jul 23 11:36:15.668741 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:15.668674    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-l6jwv"]
Jul 23 11:36:15.668901 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:15.668799    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l6jwv"
Jul 23 11:36:15.668962 ip-10-0-141-229 kubenswrapper[2563]: E0723 11:36:15.668896    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l6jwv" podUID="b30ea3bc-3059-4d68-934c-32cde3d06fbb"
Jul 23 11:36:15.669340 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:15.669319    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-cvbbt"]
Jul 23 11:36:15.669418 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:15.669404    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cvbbt"
Jul 23 11:36:15.669492 ip-10-0-141-229 kubenswrapper[2563]: E0723 11:36:15.669475    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cvbbt" podUID="7a0946cf-0f04-49d5-ab74-6f64a48f8e5d"
Jul 23 11:36:16.297220 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:16.297186    2563 generic.go:356] "Generic (PLEG): container finished" podID="7a2f8e1f-040a-4e01-88a7-9909e2fbc8b3" containerID="8ee5f235fb46c0892a3eeccc188721ef562ab6d7f60fc215c13423b978140dcb" exitCode=0
Jul 23 11:36:16.297655 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:16.297273    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-prdbw" event={"ID":"7a2f8e1f-040a-4e01-88a7-9909e2fbc8b3","Type":"ContainerDied","Data":"8ee5f235fb46c0892a3eeccc188721ef562ab6d7f60fc215c13423b978140dcb"}
Jul 23 11:36:16.886497 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:16.884754    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:36:16.963074 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:16.962984    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:36:17.072403 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:17.072375    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pbmtq"
Jul 23 11:36:17.072537 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:17.072418    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l6jwv"
Jul 23 11:36:17.072537 ip-10-0-141-229 kubenswrapper[2563]: E0723 11:36:17.072485    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pbmtq" podUID="cbdff1cb-47fd-4ed9-bec9-419ea8da3aaa"
Jul 23 11:36:17.072632 ip-10-0-141-229 kubenswrapper[2563]: E0723 11:36:17.072597    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l6jwv" podUID="b30ea3bc-3059-4d68-934c-32cde3d06fbb"
Jul 23 11:36:18.009005 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:18.008922    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:36:18.071928 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:18.071897    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cvbbt"
Jul 23 11:36:18.072090 ip-10-0-141-229 kubenswrapper[2563]: E0723 11:36:18.072015    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cvbbt" podUID="7a0946cf-0f04-49d5-ab74-6f64a48f8e5d"
Jul 23 11:36:18.072711 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:18.072669    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:36:18.181643 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:18.181609    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:36:18.242264 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:18.242232    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:36:18.570945 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:18.570907    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:36:18.649663 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:18.649579    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:36:18.722011 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:18.721976    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:36:18.781584 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:18.781548    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:36:18.801546 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:18.801508    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7a0946cf-0f04-49d5-ab74-6f64a48f8e5d-original-pull-secret\") pod \"global-pull-secret-syncer-cvbbt\" (UID: \"7a0946cf-0f04-49d5-ab74-6f64a48f8e5d\") " pod="kube-system/global-pull-secret-syncer-cvbbt"
Jul 23 11:36:18.801653 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:18.801559    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-528jd\" (UniqueName: \"kubernetes.io/projected/b30ea3bc-3059-4d68-934c-32cde3d06fbb-kube-api-access-528jd\") pod \"network-check-target-l6jwv\" (UID: \"b30ea3bc-3059-4d68-934c-32cde3d06fbb\") " pod="openshift-network-diagnostics/network-check-target-l6jwv"
Jul 23 11:36:18.801653 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:18.801585    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cbdff1cb-47fd-4ed9-bec9-419ea8da3aaa-metrics-certs\") pod \"network-metrics-daemon-pbmtq\" (UID: \"cbdff1cb-47fd-4ed9-bec9-419ea8da3aaa\") " pod="openshift-multus/network-metrics-daemon-pbmtq"
Jul 23 11:36:18.801757 ip-10-0-141-229 kubenswrapper[2563]: E0723 11:36:18.801705    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 23 11:36:18.801757 ip-10-0-141-229 kubenswrapper[2563]: E0723 11:36:18.801724    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 23 11:36:18.801757 ip-10-0-141-229 kubenswrapper[2563]: E0723 11:36:18.801741    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 23 11:36:18.801757 ip-10-0-141-229 kubenswrapper[2563]: E0723 11:36:18.801753    2563 projected.go:196] Error preparing data for projected volume kube-api-access-528jd for pod openshift-network-diagnostics/network-check-target-l6jwv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 23 11:36:18.801883 ip-10-0-141-229 kubenswrapper[2563]: E0723 11:36:18.801778    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7a0946cf-0f04-49d5-ab74-6f64a48f8e5d-original-pull-secret podName:7a0946cf-0f04-49d5-ab74-6f64a48f8e5d nodeName:}" failed. No retries permitted until 2026-07-23 11:36:50.801757697 +0000 UTC m=+65.194695483 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7a0946cf-0f04-49d5-ab74-6f64a48f8e5d-original-pull-secret") pod "global-pull-secret-syncer-cvbbt" (UID: "7a0946cf-0f04-49d5-ab74-6f64a48f8e5d") : object "kube-system"/"original-pull-secret" not registered
Jul 23 11:36:18.801883 ip-10-0-141-229 kubenswrapper[2563]: E0723 11:36:18.801795    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b30ea3bc-3059-4d68-934c-32cde3d06fbb-kube-api-access-528jd podName:b30ea3bc-3059-4d68-934c-32cde3d06fbb nodeName:}" failed. No retries permitted until 2026-07-23 11:36:50.801785539 +0000 UTC m=+65.194723322 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-528jd" (UniqueName: "kubernetes.io/projected/b30ea3bc-3059-4d68-934c-32cde3d06fbb-kube-api-access-528jd") pod "network-check-target-l6jwv" (UID: "b30ea3bc-3059-4d68-934c-32cde3d06fbb") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 23 11:36:18.801883 ip-10-0-141-229 kubenswrapper[2563]: E0723 11:36:18.801705    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 23 11:36:18.801883 ip-10-0-141-229 kubenswrapper[2563]: E0723 11:36:18.801861    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cbdff1cb-47fd-4ed9-bec9-419ea8da3aaa-metrics-certs podName:cbdff1cb-47fd-4ed9-bec9-419ea8da3aaa nodeName:}" failed. No retries permitted until 2026-07-23 11:36:50.801836483 +0000 UTC m=+65.194774270 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cbdff1cb-47fd-4ed9-bec9-419ea8da3aaa-metrics-certs") pod "network-metrics-daemon-pbmtq" (UID: "cbdff1cb-47fd-4ed9-bec9-419ea8da3aaa") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 23 11:36:19.072047 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:19.071956    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pbmtq"
Jul 23 11:36:19.072392 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:19.071971    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l6jwv"
Jul 23 11:36:19.072392 ip-10-0-141-229 kubenswrapper[2563]: E0723 11:36:19.072101    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pbmtq" podUID="cbdff1cb-47fd-4ed9-bec9-419ea8da3aaa"
Jul 23 11:36:19.072392 ip-10-0-141-229 kubenswrapper[2563]: E0723 11:36:19.072126    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l6jwv" podUID="b30ea3bc-3059-4d68-934c-32cde3d06fbb"
Jul 23 11:36:19.162555 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:19.162524    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-dmnft_5ef3b70f-a117-4dcd-9380-5c5d36e596c0/dns-node-resolver/0.log"
Jul 23 11:36:19.306783 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:19.306747    2563 generic.go:356] "Generic (PLEG): container finished" podID="7a2f8e1f-040a-4e01-88a7-9909e2fbc8b3" containerID="982bea21185c51dad6b835212d7b4aaa9711328b64b5569299694f7d5f9e7243" exitCode=0
Jul 23 11:36:19.306938 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:19.306808    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-prdbw" event={"ID":"7a2f8e1f-040a-4e01-88a7-9909e2fbc8b3","Type":"ContainerDied","Data":"982bea21185c51dad6b835212d7b4aaa9711328b64b5569299694f7d5f9e7243"}
Jul 23 11:36:20.072423 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:20.072389    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cvbbt"
Jul 23 11:36:20.072923 ip-10-0-141-229 kubenswrapper[2563]: E0723 11:36:20.072520    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cvbbt" podUID="7a0946cf-0f04-49d5-ab74-6f64a48f8e5d"
Jul 23 11:36:20.422499 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:20.422462    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:36:20.472162 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:20.472128    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-229.ec2.internal" event="NodeReady"
Jul 23 11:36:20.472340 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:20.472302    2563 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 23 11:36:20.546894 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:20.546859    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-668768fb6b-zlz7l"]
Jul 23 11:36:20.549051 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:20.549025    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-668768fb6b-zlz7l"
Jul 23 11:36:20.554295 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:20.554273    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-pmn7v\""
Jul 23 11:36:20.554295 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:20.554278    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 23 11:36:20.554495 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:20.554339    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 23 11:36:20.554599 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:20.554581    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 23 11:36:20.571316 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:20.571287    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-d5p84"]
Jul 23 11:36:20.572909 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:20.572871    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 23 11:36:20.573178 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:20.573161    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-d5p84"
Jul 23 11:36:20.575889 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:20.575869    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 23 11:36:20.576174 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:20.576146    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 23 11:36:20.576174 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:20.576164    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-pb4w9\""
Jul 23 11:36:20.576325 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:20.576147    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 23 11:36:20.576436 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:20.576420    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-dg4zg_2ba42487-6ccf-4e15-93df-65f0623c41ba/node-ca/0.log"
Jul 23 11:36:20.583743 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:20.581990    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-668768fb6b-zlz7l"]
Jul 23 11:36:20.583743 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:20.582841    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-r4f2b"]
Jul 23 11:36:20.584669 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:20.584649    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-r4f2b"
Jul 23 11:36:20.587755 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:20.587737    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 23 11:36:20.587903 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:20.587884    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-jxxcm\""
Jul 23 11:36:20.588173 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:20.588151    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 23 11:36:20.588467 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:20.588449    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 23 11:36:20.589042 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:20.589023    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 23 11:36:20.599202 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:20.599179    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-t2l85"]
Jul 23 11:36:20.601671 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:20.601600    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-t2l85"
Jul 23 11:36:20.601960 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:20.601797    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:36:20.603650 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:20.603628    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-d5p84"]
Jul 23 11:36:20.603851 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:20.603832    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-r4f2b"]
Jul 23 11:36:20.605802 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:20.605530    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 23 11:36:20.605802 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:20.605795    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 23 11:36:20.605935 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:20.605819    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-67ffh\""
Jul 23 11:36:20.627404 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:20.627371    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-t2l85"]
Jul 23 11:36:20.715867 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:20.715831    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/94d3e57f-8c9b-48c6-9566-8998f808f363-registry-certificates\") pod \"image-registry-668768fb6b-zlz7l\" (UID: \"94d3e57f-8c9b-48c6-9566-8998f808f363\") " pod="openshift-image-registry/image-registry-668768fb6b-zlz7l"
Jul 23 11:36:20.716050 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:20.715905    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/867645c0-1161-4522-b68f-f597f275ed37-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-r4f2b\" (UID: \"867645c0-1161-4522-b68f-f597f275ed37\") " pod="openshift-insights/insights-runtime-extractor-r4f2b"
Jul 23 11:36:20.716050 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:20.715966    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/94d3e57f-8c9b-48c6-9566-8998f808f363-ca-trust-extracted\") pod \"image-registry-668768fb6b-zlz7l\" (UID: \"94d3e57f-8c9b-48c6-9566-8998f808f363\") " pod="openshift-image-registry/image-registry-668768fb6b-zlz7l"
Jul 23 11:36:20.716050 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:20.715994    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/94d3e57f-8c9b-48c6-9566-8998f808f363-image-registry-private-configuration\") pod \"image-registry-668768fb6b-zlz7l\" (UID: \"94d3e57f-8c9b-48c6-9566-8998f808f363\") " pod="openshift-image-registry/image-registry-668768fb6b-zlz7l"
Jul 23 11:36:20.716050 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:20.716030    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/804c9466-58e2-4986-b0bf-2e85cba3d0e4-metrics-tls\") pod \"dns-default-t2l85\" (UID: \"804c9466-58e2-4986-b0bf-2e85cba3d0e4\") " pod="openshift-dns/dns-default-t2l85"
Jul 23 11:36:20.716255 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:20.716122    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/804c9466-58e2-4986-b0bf-2e85cba3d0e4-config-volume\") pod \"dns-default-t2l85\" (UID: \"804c9466-58e2-4986-b0bf-2e85cba3d0e4\") " pod="openshift-dns/dns-default-t2l85"
Jul 23 11:36:20.716255 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:20.716167    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/804c9466-58e2-4986-b0bf-2e85cba3d0e4-tmp-dir\") pod \"dns-default-t2l85\" (UID: \"804c9466-58e2-4986-b0bf-2e85cba3d0e4\") " pod="openshift-dns/dns-default-t2l85"
Jul 23 11:36:20.716255 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:20.716201    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/595ac104-73d9-408b-b1e4-923312a4f5e7-cert\") pod \"ingress-canary-d5p84\" (UID: \"595ac104-73d9-408b-b1e4-923312a4f5e7\") " pod="openshift-ingress-canary/ingress-canary-d5p84"
Jul 23 11:36:20.716255 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:20.716232    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/867645c0-1161-4522-b68f-f597f275ed37-data-volume\") pod \"insights-runtime-extractor-r4f2b\" (UID: \"867645c0-1161-4522-b68f-f597f275ed37\") " pod="openshift-insights/insights-runtime-extractor-r4f2b"
Jul 23 11:36:20.716422 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:20.716260    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/94d3e57f-8c9b-48c6-9566-8998f808f363-trusted-ca\") pod \"image-registry-668768fb6b-zlz7l\" (UID: \"94d3e57f-8c9b-48c6-9566-8998f808f363\") " pod="openshift-image-registry/image-registry-668768fb6b-zlz7l"
Jul 23 11:36:20.716422 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:20.716346    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9thnw\" (UniqueName: \"kubernetes.io/projected/804c9466-58e2-4986-b0bf-2e85cba3d0e4-kube-api-access-9thnw\") pod \"dns-default-t2l85\" (UID: \"804c9466-58e2-4986-b0bf-2e85cba3d0e4\") " pod="openshift-dns/dns-default-t2l85"
Jul 23 11:36:20.716422 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:20.716376    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/94d3e57f-8c9b-48c6-9566-8998f808f363-registry-tls\") pod \"image-registry-668768fb6b-zlz7l\" (UID: \"94d3e57f-8c9b-48c6-9566-8998f808f363\") " pod="openshift-image-registry/image-registry-668768fb6b-zlz7l"
Jul 23 11:36:20.716422 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:20.716403    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tngqz\" (UniqueName: \"kubernetes.io/projected/595ac104-73d9-408b-b1e4-923312a4f5e7-kube-api-access-tngqz\") pod \"ingress-canary-d5p84\" (UID: \"595ac104-73d9-408b-b1e4-923312a4f5e7\") " pod="openshift-ingress-canary/ingress-canary-d5p84"
Jul 23 11:36:20.716599 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:20.716436    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kmzx8\" (UniqueName: \"kubernetes.io/projected/867645c0-1161-4522-b68f-f597f275ed37-kube-api-access-kmzx8\") pod \"insights-runtime-extractor-r4f2b\" (UID: \"867645c0-1161-4522-b68f-f597f275ed37\") " pod="openshift-insights/insights-runtime-extractor-r4f2b"
Jul 23 11:36:20.716599 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:20.716516    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/94d3e57f-8c9b-48c6-9566-8998f808f363-installation-pull-secrets\") pod \"image-registry-668768fb6b-zlz7l\" (UID: \"94d3e57f-8c9b-48c6-9566-8998f808f363\") " pod="openshift-image-registry/image-registry-668768fb6b-zlz7l"
Jul 23 11:36:20.716599 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:20.716543    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/94d3e57f-8c9b-48c6-9566-8998f808f363-bound-sa-token\") pod \"image-registry-668768fb6b-zlz7l\" (UID: \"94d3e57f-8c9b-48c6-9566-8998f808f363\") " pod="openshift-image-registry/image-registry-668768fb6b-zlz7l"
Jul 23 11:36:20.716599 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:20.716587    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmwqd\" (UniqueName: \"kubernetes.io/projected/94d3e57f-8c9b-48c6-9566-8998f808f363-kube-api-access-mmwqd\") pod \"image-registry-668768fb6b-zlz7l\" (UID: \"94d3e57f-8c9b-48c6-9566-8998f808f363\") " pod="openshift-image-registry/image-registry-668768fb6b-zlz7l"
Jul 23 11:36:20.716771 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:20.716618    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/867645c0-1161-4522-b68f-f597f275ed37-crio-socket\") pod \"insights-runtime-extractor-r4f2b\" (UID: \"867645c0-1161-4522-b68f-f597f275ed37\") " pod="openshift-insights/insights-runtime-extractor-r4f2b"
Jul 23 11:36:20.716771 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:20.716638    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/867645c0-1161-4522-b68f-f597f275ed37-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-r4f2b\" (UID: \"867645c0-1161-4522-b68f-f597f275ed37\") " pod="openshift-insights/insights-runtime-extractor-r4f2b"
Jul 23 11:36:20.817886 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:20.817850    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/804c9466-58e2-4986-b0bf-2e85cba3d0e4-metrics-tls\") pod \"dns-default-t2l85\" (UID: \"804c9466-58e2-4986-b0bf-2e85cba3d0e4\") " pod="openshift-dns/dns-default-t2l85"
Jul 23 11:36:20.818090 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:20.817895    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/804c9466-58e2-4986-b0bf-2e85cba3d0e4-config-volume\") pod \"dns-default-t2l85\" (UID: \"804c9466-58e2-4986-b0bf-2e85cba3d0e4\") " pod="openshift-dns/dns-default-t2l85"
Jul 23 11:36:20.818090 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:20.817920    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/804c9466-58e2-4986-b0bf-2e85cba3d0e4-tmp-dir\") pod \"dns-default-t2l85\" (UID: \"804c9466-58e2-4986-b0bf-2e85cba3d0e4\") " pod="openshift-dns/dns-default-t2l85"
Jul 23 11:36:20.818090 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:20.817942    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/595ac104-73d9-408b-b1e4-923312a4f5e7-cert\") pod \"ingress-canary-d5p84\" (UID: \"595ac104-73d9-408b-b1e4-923312a4f5e7\") " pod="openshift-ingress-canary/ingress-canary-d5p84"
Jul 23 11:36:20.818090 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:20.817966    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/867645c0-1161-4522-b68f-f597f275ed37-data-volume\") pod \"insights-runtime-extractor-r4f2b\" (UID: \"867645c0-1161-4522-b68f-f597f275ed37\") " pod="openshift-insights/insights-runtime-extractor-r4f2b"
Jul 23 11:36:20.818090 ip-10-0-141-229 kubenswrapper[2563]: E0723 11:36:20.818026    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 23 11:36:20.818343 ip-10-0-141-229 kubenswrapper[2563]: E0723 11:36:20.818103    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/804c9466-58e2-4986-b0bf-2e85cba3d0e4-metrics-tls podName:804c9466-58e2-4986-b0bf-2e85cba3d0e4 nodeName:}" failed. No retries permitted until 2026-07-23 11:36:21.318084724 +0000 UTC m=+35.711022510 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/804c9466-58e2-4986-b0bf-2e85cba3d0e4-metrics-tls") pod "dns-default-t2l85" (UID: "804c9466-58e2-4986-b0bf-2e85cba3d0e4") : secret "dns-default-metrics-tls" not found
Jul 23 11:36:20.818343 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:20.818288    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/94d3e57f-8c9b-48c6-9566-8998f808f363-trusted-ca\") pod \"image-registry-668768fb6b-zlz7l\" (UID: \"94d3e57f-8c9b-48c6-9566-8998f808f363\") " pod="openshift-image-registry/image-registry-668768fb6b-zlz7l"
Jul 23 11:36:20.818343 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:20.818314    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/804c9466-58e2-4986-b0bf-2e85cba3d0e4-tmp-dir\") pod \"dns-default-t2l85\" (UID: \"804c9466-58e2-4986-b0bf-2e85cba3d0e4\") " pod="openshift-dns/dns-default-t2l85"
Jul 23 11:36:20.818343 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:20.818337    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/867645c0-1161-4522-b68f-f597f275ed37-data-volume\") pod \"insights-runtime-extractor-r4f2b\" (UID: \"867645c0-1161-4522-b68f-f597f275ed37\") " pod="openshift-insights/insights-runtime-extractor-r4f2b"
Jul 23 11:36:20.818538 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:20.818368    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9thnw\" (UniqueName: \"kubernetes.io/projected/804c9466-58e2-4986-b0bf-2e85cba3d0e4-kube-api-access-9thnw\") pod \"dns-default-t2l85\" (UID: \"804c9466-58e2-4986-b0bf-2e85cba3d0e4\") " pod="openshift-dns/dns-default-t2l85"
Jul 23 11:36:20.818538 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:20.818388    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/94d3e57f-8c9b-48c6-9566-8998f808f363-registry-tls\") pod \"image-registry-668768fb6b-zlz7l\" (UID: \"94d3e57f-8c9b-48c6-9566-8998f808f363\") " pod="openshift-image-registry/image-registry-668768fb6b-zlz7l"
Jul 23 11:36:20.818538 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:20.818407    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tngqz\" (UniqueName: \"kubernetes.io/projected/595ac104-73d9-408b-b1e4-923312a4f5e7-kube-api-access-tngqz\") pod \"ingress-canary-d5p84\" (UID: \"595ac104-73d9-408b-b1e4-923312a4f5e7\") " pod="openshift-ingress-canary/ingress-canary-d5p84"
Jul 23 11:36:20.818538 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:20.818431    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kmzx8\" (UniqueName: \"kubernetes.io/projected/867645c0-1161-4522-b68f-f597f275ed37-kube-api-access-kmzx8\") pod \"insights-runtime-extractor-r4f2b\" (UID: \"867645c0-1161-4522-b68f-f597f275ed37\") " pod="openshift-insights/insights-runtime-extractor-r4f2b"
Jul 23 11:36:20.818538 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:20.818473    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/94d3e57f-8c9b-48c6-9566-8998f808f363-installation-pull-secrets\") pod \"image-registry-668768fb6b-zlz7l\" (UID: \"94d3e57f-8c9b-48c6-9566-8998f808f363\") " pod="openshift-image-registry/image-registry-668768fb6b-zlz7l"
Jul 23 11:36:20.818538 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:20.818496    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/94d3e57f-8c9b-48c6-9566-8998f808f363-bound-sa-token\") pod \"image-registry-668768fb6b-zlz7l\" (UID: \"94d3e57f-8c9b-48c6-9566-8998f808f363\") " pod="openshift-image-registry/image-registry-668768fb6b-zlz7l"
Jul 23 11:36:20.818538 ip-10-0-141-229 kubenswrapper[2563]: E0723 11:36:20.818501    2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 23 11:36:20.818538 ip-10-0-141-229 kubenswrapper[2563]: E0723 11:36:20.818519    2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-668768fb6b-zlz7l: secret "image-registry-tls" not found
Jul 23 11:36:20.818932 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:20.818546    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/804c9466-58e2-4986-b0bf-2e85cba3d0e4-config-volume\") pod \"dns-default-t2l85\" (UID: \"804c9466-58e2-4986-b0bf-2e85cba3d0e4\") " pod="openshift-dns/dns-default-t2l85"
Jul 23 11:36:20.818932 ip-10-0-141-229 kubenswrapper[2563]: E0723 11:36:20.818567    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/94d3e57f-8c9b-48c6-9566-8998f808f363-registry-tls podName:94d3e57f-8c9b-48c6-9566-8998f808f363 nodeName:}" failed. No retries permitted until 2026-07-23 11:36:21.318551704 +0000 UTC m=+35.711489470 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/94d3e57f-8c9b-48c6-9566-8998f808f363-registry-tls") pod "image-registry-668768fb6b-zlz7l" (UID: "94d3e57f-8c9b-48c6-9566-8998f808f363") : secret "image-registry-tls" not found
Jul 23 11:36:20.818932 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:20.818517    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmwqd\" (UniqueName: \"kubernetes.io/projected/94d3e57f-8c9b-48c6-9566-8998f808f363-kube-api-access-mmwqd\") pod \"image-registry-668768fb6b-zlz7l\" (UID: \"94d3e57f-8c9b-48c6-9566-8998f808f363\") " pod="openshift-image-registry/image-registry-668768fb6b-zlz7l"
Jul 23 11:36:20.818932 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:20.818708    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/867645c0-1161-4522-b68f-f597f275ed37-crio-socket\") pod \"insights-runtime-extractor-r4f2b\" (UID: \"867645c0-1161-4522-b68f-f597f275ed37\") " pod="openshift-insights/insights-runtime-extractor-r4f2b"
Jul 23 11:36:20.818932 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:20.818740    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/867645c0-1161-4522-b68f-f597f275ed37-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-r4f2b\" (UID: \"867645c0-1161-4522-b68f-f597f275ed37\") " pod="openshift-insights/insights-runtime-extractor-r4f2b"
Jul 23 11:36:20.818932 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:20.818774    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/94d3e57f-8c9b-48c6-9566-8998f808f363-registry-certificates\") pod \"image-registry-668768fb6b-zlz7l\" (UID: \"94d3e57f-8c9b-48c6-9566-8998f808f363\") " pod="openshift-image-registry/image-registry-668768fb6b-zlz7l"
Jul 23 11:36:20.818932 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:20.818802    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/867645c0-1161-4522-b68f-f597f275ed37-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-r4f2b\" (UID: \"867645c0-1161-4522-b68f-f597f275ed37\") " pod="openshift-insights/insights-runtime-extractor-r4f2b"
Jul 23 11:36:20.818932 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:20.818832    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/94d3e57f-8c9b-48c6-9566-8998f808f363-ca-trust-extracted\") pod \"image-registry-668768fb6b-zlz7l\" (UID: \"94d3e57f-8c9b-48c6-9566-8998f808f363\") " pod="openshift-image-registry/image-registry-668768fb6b-zlz7l"
Jul 23 11:36:20.818932 ip-10-0-141-229 kubenswrapper[2563]: E0723 11:36:20.818840    2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 23 11:36:20.818932 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:20.818860    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/94d3e57f-8c9b-48c6-9566-8998f808f363-image-registry-private-configuration\") pod \"image-registry-668768fb6b-zlz7l\" (UID: \"94d3e57f-8c9b-48c6-9566-8998f808f363\") " pod="openshift-image-registry/image-registry-668768fb6b-zlz7l"
Jul 23 11:36:20.818932 ip-10-0-141-229 kubenswrapper[2563]: E0723 11:36:20.818905    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/867645c0-1161-4522-b68f-f597f275ed37-kube-rbac-proxy-cm podName:867645c0-1161-4522-b68f-f597f275ed37 nodeName:}" failed. No retries permitted until 2026-07-23 11:36:21.318889589 +0000 UTC m=+35.711827357 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/867645c0-1161-4522-b68f-f597f275ed37-kube-rbac-proxy-cm") pod "insights-runtime-extractor-r4f2b" (UID: "867645c0-1161-4522-b68f-f597f275ed37") : configmap "kube-rbac-proxy" not found
Jul 23 11:36:20.818932 ip-10-0-141-229 kubenswrapper[2563]: E0723 11:36:20.818919    2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 23 11:36:20.819495 ip-10-0-141-229 kubenswrapper[2563]: E0723 11:36:20.818967    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/867645c0-1161-4522-b68f-f597f275ed37-insights-runtime-extractor-tls podName:867645c0-1161-4522-b68f-f597f275ed37 nodeName:}" failed. No retries permitted until 2026-07-23 11:36:21.318955493 +0000 UTC m=+35.711893259 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/867645c0-1161-4522-b68f-f597f275ed37-insights-runtime-extractor-tls") pod "insights-runtime-extractor-r4f2b" (UID: "867645c0-1161-4522-b68f-f597f275ed37") : secret "insights-runtime-extractor-tls" not found
Jul 23 11:36:20.819495 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:20.818913    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/867645c0-1161-4522-b68f-f597f275ed37-crio-socket\") pod \"insights-runtime-extractor-r4f2b\" (UID: \"867645c0-1161-4522-b68f-f597f275ed37\") " pod="openshift-insights/insights-runtime-extractor-r4f2b"
Jul 23 11:36:20.819495 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:20.819225    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/94d3e57f-8c9b-48c6-9566-8998f808f363-ca-trust-extracted\") pod \"image-registry-668768fb6b-zlz7l\" (UID: \"94d3e57f-8c9b-48c6-9566-8998f808f363\") " pod="openshift-image-registry/image-registry-668768fb6b-zlz7l"
Jul 23 11:36:20.819495 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:20.819352    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/94d3e57f-8c9b-48c6-9566-8998f808f363-trusted-ca\") pod \"image-registry-668768fb6b-zlz7l\" (UID: \"94d3e57f-8c9b-48c6-9566-8998f808f363\") " pod="openshift-image-registry/image-registry-668768fb6b-zlz7l"
Jul 23 11:36:20.819495 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:20.819453    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/94d3e57f-8c9b-48c6-9566-8998f808f363-registry-certificates\") pod \"image-registry-668768fb6b-zlz7l\" (UID: \"94d3e57f-8c9b-48c6-9566-8998f808f363\") " pod="openshift-image-registry/image-registry-668768fb6b-zlz7l"
Jul 23 11:36:20.823978 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:20.823952    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/94d3e57f-8c9b-48c6-9566-8998f808f363-installation-pull-secrets\") pod \"image-registry-668768fb6b-zlz7l\" (UID: \"94d3e57f-8c9b-48c6-9566-8998f808f363\") " pod="openshift-image-registry/image-registry-668768fb6b-zlz7l"
Jul 23 11:36:20.824100 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:20.823961    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/94d3e57f-8c9b-48c6-9566-8998f808f363-image-registry-private-configuration\") pod \"image-registry-668768fb6b-zlz7l\" (UID: \"94d3e57f-8c9b-48c6-9566-8998f808f363\") " pod="openshift-image-registry/image-registry-668768fb6b-zlz7l"
Jul 23 11:36:20.824100 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:20.824035    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/595ac104-73d9-408b-b1e4-923312a4f5e7-cert\") pod \"ingress-canary-d5p84\" (UID: \"595ac104-73d9-408b-b1e4-923312a4f5e7\") " pod="openshift-ingress-canary/ingress-canary-d5p84"
Jul 23 11:36:20.829870 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:20.829840    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/94d3e57f-8c9b-48c6-9566-8998f808f363-bound-sa-token\") pod \"image-registry-668768fb6b-zlz7l\" (UID: \"94d3e57f-8c9b-48c6-9566-8998f808f363\") " pod="openshift-image-registry/image-registry-668768fb6b-zlz7l"
Jul 23 11:36:20.830310 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:20.830287    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kmzx8\" (UniqueName: \"kubernetes.io/projected/867645c0-1161-4522-b68f-f597f275ed37-kube-api-access-kmzx8\") pod \"insights-runtime-extractor-r4f2b\" (UID: \"867645c0-1161-4522-b68f-f597f275ed37\") " pod="openshift-insights/insights-runtime-extractor-r4f2b"
Jul 23 11:36:20.830411 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:20.830390    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmwqd\" (UniqueName: \"kubernetes.io/projected/94d3e57f-8c9b-48c6-9566-8998f808f363-kube-api-access-mmwqd\") pod \"image-registry-668768fb6b-zlz7l\" (UID: \"94d3e57f-8c9b-48c6-9566-8998f808f363\") " pod="openshift-image-registry/image-registry-668768fb6b-zlz7l"
Jul 23 11:36:20.832542 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:20.830574    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tngqz\" (UniqueName: \"kubernetes.io/projected/595ac104-73d9-408b-b1e4-923312a4f5e7-kube-api-access-tngqz\") pod \"ingress-canary-d5p84\" (UID: \"595ac104-73d9-408b-b1e4-923312a4f5e7\") " pod="openshift-ingress-canary/ingress-canary-d5p84"
Jul 23 11:36:20.832542 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:20.830658    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9thnw\" (UniqueName: \"kubernetes.io/projected/804c9466-58e2-4986-b0bf-2e85cba3d0e4-kube-api-access-9thnw\") pod \"dns-default-t2l85\" (UID: \"804c9466-58e2-4986-b0bf-2e85cba3d0e4\") " pod="openshift-dns/dns-default-t2l85"
Jul 23 11:36:20.867951 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:20.867915    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:36:20.884654 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:20.884620    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-d5p84"
Jul 23 11:36:20.967420 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:20.967198    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:36:21.051907 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:21.051871    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-d5p84"]
Jul 23 11:36:21.057528 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:36:21.057493    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod595ac104_73d9_408b_b1e4_923312a4f5e7.slice/crio-c3d630f77ae0bf0808d0b0a65945d75b83ef7ab8b7a96eb60bb9b2f6c6eff8ea WatchSource:0}: Error finding container c3d630f77ae0bf0808d0b0a65945d75b83ef7ab8b7a96eb60bb9b2f6c6eff8ea: Status 404 returned error can't find the container with id c3d630f77ae0bf0808d0b0a65945d75b83ef7ab8b7a96eb60bb9b2f6c6eff8ea
Jul 23 11:36:21.072413 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:21.072377    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pbmtq"
Jul 23 11:36:21.072535 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:21.072387    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l6jwv"
Jul 23 11:36:21.076736 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:21.076489    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-7flkj\""
Jul 23 11:36:21.076736 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:21.076553    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-xtz4b\""
Jul 23 11:36:21.076736 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:21.076603    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 23 11:36:21.076956 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:21.076844    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 23 11:36:21.076956 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:21.076851    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 23 11:36:21.313674 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:21.313639    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-d5p84" event={"ID":"595ac104-73d9-408b-b1e4-923312a4f5e7","Type":"ContainerStarted","Data":"c3d630f77ae0bf0808d0b0a65945d75b83ef7ab8b7a96eb60bb9b2f6c6eff8ea"}
Jul 23 11:36:21.323036 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:21.322999    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/804c9466-58e2-4986-b0bf-2e85cba3d0e4-metrics-tls\") pod \"dns-default-t2l85\" (UID: \"804c9466-58e2-4986-b0bf-2e85cba3d0e4\") " pod="openshift-dns/dns-default-t2l85"
Jul 23 11:36:21.323175 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:21.323072    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/94d3e57f-8c9b-48c6-9566-8998f808f363-registry-tls\") pod \"image-registry-668768fb6b-zlz7l\" (UID: \"94d3e57f-8c9b-48c6-9566-8998f808f363\") " pod="openshift-image-registry/image-registry-668768fb6b-zlz7l"
Jul 23 11:36:21.323175 ip-10-0-141-229 kubenswrapper[2563]: E0723 11:36:21.323167    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 23 11:36:21.323299 ip-10-0-141-229 kubenswrapper[2563]: E0723 11:36:21.323170    2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 23 11:36:21.323299 ip-10-0-141-229 kubenswrapper[2563]: E0723 11:36:21.323188    2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-668768fb6b-zlz7l: secret "image-registry-tls" not found
Jul 23 11:36:21.323299 ip-10-0-141-229 kubenswrapper[2563]: E0723 11:36:21.323225    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/94d3e57f-8c9b-48c6-9566-8998f808f363-registry-tls podName:94d3e57f-8c9b-48c6-9566-8998f808f363 nodeName:}" failed. No retries permitted until 2026-07-23 11:36:22.323209097 +0000 UTC m=+36.716146864 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/94d3e57f-8c9b-48c6-9566-8998f808f363-registry-tls") pod "image-registry-668768fb6b-zlz7l" (UID: "94d3e57f-8c9b-48c6-9566-8998f808f363") : secret "image-registry-tls" not found
Jul 23 11:36:21.323299 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:21.323266    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/867645c0-1161-4522-b68f-f597f275ed37-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-r4f2b\" (UID: \"867645c0-1161-4522-b68f-f597f275ed37\") " pod="openshift-insights/insights-runtime-extractor-r4f2b"
Jul 23 11:36:21.323492 ip-10-0-141-229 kubenswrapper[2563]: E0723 11:36:21.323334    2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 23 11:36:21.323492 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:21.323341    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/867645c0-1161-4522-b68f-f597f275ed37-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-r4f2b\" (UID: \"867645c0-1161-4522-b68f-f597f275ed37\") " pod="openshift-insights/insights-runtime-extractor-r4f2b"
Jul 23 11:36:21.323492 ip-10-0-141-229 kubenswrapper[2563]: E0723 11:36:21.323393    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/867645c0-1161-4522-b68f-f597f275ed37-kube-rbac-proxy-cm podName:867645c0-1161-4522-b68f-f597f275ed37 nodeName:}" failed. No retries permitted until 2026-07-23 11:36:22.323374508 +0000 UTC m=+36.716312288 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/867645c0-1161-4522-b68f-f597f275ed37-kube-rbac-proxy-cm") pod "insights-runtime-extractor-r4f2b" (UID: "867645c0-1161-4522-b68f-f597f275ed37") : configmap "kube-rbac-proxy" not found
Jul 23 11:36:21.323492 ip-10-0-141-229 kubenswrapper[2563]: E0723 11:36:21.323419    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/804c9466-58e2-4986-b0bf-2e85cba3d0e4-metrics-tls podName:804c9466-58e2-4986-b0bf-2e85cba3d0e4 nodeName:}" failed. No retries permitted until 2026-07-23 11:36:22.32340913 +0000 UTC m=+36.716346899 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/804c9466-58e2-4986-b0bf-2e85cba3d0e4-metrics-tls") pod "dns-default-t2l85" (UID: "804c9466-58e2-4986-b0bf-2e85cba3d0e4") : secret "dns-default-metrics-tls" not found
Jul 23 11:36:21.323492 ip-10-0-141-229 kubenswrapper[2563]: E0723 11:36:21.323466    2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 23 11:36:21.323677 ip-10-0-141-229 kubenswrapper[2563]: E0723 11:36:21.323523    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/867645c0-1161-4522-b68f-f597f275ed37-insights-runtime-extractor-tls podName:867645c0-1161-4522-b68f-f597f275ed37 nodeName:}" failed. No retries permitted until 2026-07-23 11:36:22.323508238 +0000 UTC m=+36.716446028 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/867645c0-1161-4522-b68f-f597f275ed37-insights-runtime-extractor-tls") pod "insights-runtime-extractor-r4f2b" (UID: "867645c0-1161-4522-b68f-f597f275ed37") : secret "insights-runtime-extractor-tls" not found
Jul 23 11:36:21.866341 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:21.866301    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:36:22.072323 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:22.072288    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cvbbt"
Jul 23 11:36:22.075629 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:22.075600    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 23 11:36:22.331627 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:22.331547    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/94d3e57f-8c9b-48c6-9566-8998f808f363-registry-tls\") pod \"image-registry-668768fb6b-zlz7l\" (UID: \"94d3e57f-8c9b-48c6-9566-8998f808f363\") " pod="openshift-image-registry/image-registry-668768fb6b-zlz7l"
Jul 23 11:36:22.331627 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:22.331607    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/867645c0-1161-4522-b68f-f597f275ed37-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-r4f2b\" (UID: \"867645c0-1161-4522-b68f-f597f275ed37\") " pod="openshift-insights/insights-runtime-extractor-r4f2b"
Jul 23 11:36:22.331879 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:22.331649    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/867645c0-1161-4522-b68f-f597f275ed37-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-r4f2b\" (UID: \"867645c0-1161-4522-b68f-f597f275ed37\") " pod="openshift-insights/insights-runtime-extractor-r4f2b"
Jul 23 11:36:22.331879 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:22.331713    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/804c9466-58e2-4986-b0bf-2e85cba3d0e4-metrics-tls\") pod \"dns-default-t2l85\" (UID: \"804c9466-58e2-4986-b0bf-2e85cba3d0e4\") " pod="openshift-dns/dns-default-t2l85"
Jul 23 11:36:22.331879 ip-10-0-141-229 kubenswrapper[2563]: E0723 11:36:22.331862    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 23 11:36:22.332018 ip-10-0-141-229 kubenswrapper[2563]: E0723 11:36:22.331925    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/804c9466-58e2-4986-b0bf-2e85cba3d0e4-metrics-tls podName:804c9466-58e2-4986-b0bf-2e85cba3d0e4 nodeName:}" failed. No retries permitted until 2026-07-23 11:36:24.331904632 +0000 UTC m=+38.724842418 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/804c9466-58e2-4986-b0bf-2e85cba3d0e4-metrics-tls") pod "dns-default-t2l85" (UID: "804c9466-58e2-4986-b0bf-2e85cba3d0e4") : secret "dns-default-metrics-tls" not found
Jul 23 11:36:22.332327 ip-10-0-141-229 kubenswrapper[2563]: E0723 11:36:22.332308    2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 23 11:36:22.332327 ip-10-0-141-229 kubenswrapper[2563]: E0723 11:36:22.332328    2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-668768fb6b-zlz7l: secret "image-registry-tls" not found
Jul 23 11:36:22.332484 ip-10-0-141-229 kubenswrapper[2563]: E0723 11:36:22.332375    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/94d3e57f-8c9b-48c6-9566-8998f808f363-registry-tls podName:94d3e57f-8c9b-48c6-9566-8998f808f363 nodeName:}" failed. No retries permitted until 2026-07-23 11:36:24.332359599 +0000 UTC m=+38.725297385 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/94d3e57f-8c9b-48c6-9566-8998f808f363-registry-tls") pod "image-registry-668768fb6b-zlz7l" (UID: "94d3e57f-8c9b-48c6-9566-8998f808f363") : secret "image-registry-tls" not found
Jul 23 11:36:22.332484 ip-10-0-141-229 kubenswrapper[2563]: E0723 11:36:22.332419    2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 23 11:36:22.332484 ip-10-0-141-229 kubenswrapper[2563]: E0723 11:36:22.332451    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/867645c0-1161-4522-b68f-f597f275ed37-kube-rbac-proxy-cm podName:867645c0-1161-4522-b68f-f597f275ed37 nodeName:}" failed. No retries permitted until 2026-07-23 11:36:24.332440581 +0000 UTC m=+38.725378362 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/867645c0-1161-4522-b68f-f597f275ed37-kube-rbac-proxy-cm") pod "insights-runtime-extractor-r4f2b" (UID: "867645c0-1161-4522-b68f-f597f275ed37") : configmap "kube-rbac-proxy" not found
Jul 23 11:36:22.332638 ip-10-0-141-229 kubenswrapper[2563]: E0723 11:36:22.332514    2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 23 11:36:22.332638 ip-10-0-141-229 kubenswrapper[2563]: E0723 11:36:22.332548    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/867645c0-1161-4522-b68f-f597f275ed37-insights-runtime-extractor-tls podName:867645c0-1161-4522-b68f-f597f275ed37 nodeName:}" failed. No retries permitted until 2026-07-23 11:36:24.332537426 +0000 UTC m=+38.725475205 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/867645c0-1161-4522-b68f-f597f275ed37-insights-runtime-extractor-tls") pod "insights-runtime-extractor-r4f2b" (UID: "867645c0-1161-4522-b68f-f597f275ed37") : secret "insights-runtime-extractor-tls" not found
Jul 23 11:36:22.465835 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:22.465794    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:36:22.567601 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:22.567567    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:36:22.700627 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:22.700590    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:36:22.880583 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:22.880543    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:36:22.971734 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:22.971577    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:36:23.262381 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:23.262292    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:36:23.360617 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:23.360576    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:36:23.459795 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:23.459758    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:36:23.753572 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:23.753536    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:36:24.155417 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:24.155384    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:36:24.302260 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:24.302225    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:36:24.348162 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:24.348119    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/804c9466-58e2-4986-b0bf-2e85cba3d0e4-metrics-tls\") pod \"dns-default-t2l85\" (UID: \"804c9466-58e2-4986-b0bf-2e85cba3d0e4\") " pod="openshift-dns/dns-default-t2l85"
Jul 23 11:36:24.348357 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:24.348219    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/94d3e57f-8c9b-48c6-9566-8998f808f363-registry-tls\") pod \"image-registry-668768fb6b-zlz7l\" (UID: \"94d3e57f-8c9b-48c6-9566-8998f808f363\") " pod="openshift-image-registry/image-registry-668768fb6b-zlz7l"
Jul 23 11:36:24.348357 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:24.348264    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/867645c0-1161-4522-b68f-f597f275ed37-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-r4f2b\" (UID: \"867645c0-1161-4522-b68f-f597f275ed37\") " pod="openshift-insights/insights-runtime-extractor-r4f2b"
Jul 23 11:36:24.348357 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:24.348299    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/867645c0-1161-4522-b68f-f597f275ed37-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-r4f2b\" (UID: \"867645c0-1161-4522-b68f-f597f275ed37\") " pod="openshift-insights/insights-runtime-extractor-r4f2b"
Jul 23 11:36:24.348357 ip-10-0-141-229 kubenswrapper[2563]: E0723 11:36:24.348301    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 23 11:36:24.348505 ip-10-0-141-229 kubenswrapper[2563]: E0723 11:36:24.348355    2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 23 11:36:24.348505 ip-10-0-141-229 kubenswrapper[2563]: E0723 11:36:24.348370    2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-668768fb6b-zlz7l: secret "image-registry-tls" not found
Jul 23 11:36:24.348505 ip-10-0-141-229 kubenswrapper[2563]: E0723 11:36:24.348392    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/804c9466-58e2-4986-b0bf-2e85cba3d0e4-metrics-tls podName:804c9466-58e2-4986-b0bf-2e85cba3d0e4 nodeName:}" failed. No retries permitted until 2026-07-23 11:36:28.348369921 +0000 UTC m=+42.741307694 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/804c9466-58e2-4986-b0bf-2e85cba3d0e4-metrics-tls") pod "dns-default-t2l85" (UID: "804c9466-58e2-4986-b0bf-2e85cba3d0e4") : secret "dns-default-metrics-tls" not found
Jul 23 11:36:24.348505 ip-10-0-141-229 kubenswrapper[2563]: E0723 11:36:24.348394    2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 23 11:36:24.348505 ip-10-0-141-229 kubenswrapper[2563]: E0723 11:36:24.348415    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/94d3e57f-8c9b-48c6-9566-8998f808f363-registry-tls podName:94d3e57f-8c9b-48c6-9566-8998f808f363 nodeName:}" failed. No retries permitted until 2026-07-23 11:36:28.348401592 +0000 UTC m=+42.741339388 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/94d3e57f-8c9b-48c6-9566-8998f808f363-registry-tls") pod "image-registry-668768fb6b-zlz7l" (UID: "94d3e57f-8c9b-48c6-9566-8998f808f363") : secret "image-registry-tls" not found
Jul 23 11:36:24.348505 ip-10-0-141-229 kubenswrapper[2563]: E0723 11:36:24.348427    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/867645c0-1161-4522-b68f-f597f275ed37-insights-runtime-extractor-tls podName:867645c0-1161-4522-b68f-f597f275ed37 nodeName:}" failed. No retries permitted until 2026-07-23 11:36:28.348419857 +0000 UTC m=+42.741357623 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/867645c0-1161-4522-b68f-f597f275ed37-insights-runtime-extractor-tls") pod "insights-runtime-extractor-r4f2b" (UID: "867645c0-1161-4522-b68f-f597f275ed37") : secret "insights-runtime-extractor-tls" not found
Jul 23 11:36:24.349000 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:24.348978    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/867645c0-1161-4522-b68f-f597f275ed37-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-r4f2b\" (UID: \"867645c0-1161-4522-b68f-f597f275ed37\") " pod="openshift-insights/insights-runtime-extractor-r4f2b"
Jul 23 11:36:24.560592 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:24.560517    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:36:25.061118 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:25.060930    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:36:25.325558 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:25.325529    2563 generic.go:356] "Generic (PLEG): container finished" podID="7a2f8e1f-040a-4e01-88a7-9909e2fbc8b3" containerID="5eeaba431dfb2faf6f0f163e4eddb0a032c35d3005514c5aeeb1c773b0b23526" exitCode=0
Jul 23 11:36:25.325982 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:25.325608    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-prdbw" event={"ID":"7a2f8e1f-040a-4e01-88a7-9909e2fbc8b3","Type":"ContainerDied","Data":"5eeaba431dfb2faf6f0f163e4eddb0a032c35d3005514c5aeeb1c773b0b23526"}
Jul 23 11:36:25.327157 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:25.327133    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-d5p84" event={"ID":"595ac104-73d9-408b-b1e4-923312a4f5e7","Type":"ContainerStarted","Data":"b75a39dfc93cd589981cd03a38094b234a5f9849b6e5f55757c6a7ede51c602c"}
Jul 23 11:36:25.352290 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:25.352252    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:36:25.372399 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:25.372352    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-d5p84" podStartSLOduration=1.623972577 podStartE2EDuration="5.372335208s" totalImagesPullingTime="3.748362631s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 11:36:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:36:25.371068773 +0000 UTC m=+39.764006563" watchObservedRunningTime="2026-07-23 11:36:25.372335208 +0000 UTC m=+39.765272996"
Jul 23 11:36:25.460613 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:25.460542    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:36:25.561759 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:25.561728    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:36:25.664398 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:25.664357    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:36:25.767445 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:25.767351    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:36:25.950917 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:25.950890    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:36:26.154814 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:26.154785    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:36:26.332275 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:26.332244    2563 generic.go:356] "Generic (PLEG): container finished" podID="7a2f8e1f-040a-4e01-88a7-9909e2fbc8b3" containerID="0415c1b6c3fac7d600a7746842f735582415f7dcd249302b4ab0021cc52e362e" exitCode=0
Jul 23 11:36:26.332650 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:26.332342    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-prdbw" event={"ID":"7a2f8e1f-040a-4e01-88a7-9909e2fbc8b3","Type":"ContainerDied","Data":"0415c1b6c3fac7d600a7746842f735582415f7dcd249302b4ab0021cc52e362e"}
Jul 23 11:36:26.546983 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:26.546911    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:36:26.609932 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:26.609898    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:36:27.144025 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:27.143985    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:36:27.248428 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:27.248399    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:36:27.337708 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:27.337662    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-prdbw" event={"ID":"7a2f8e1f-040a-4e01-88a7-9909e2fbc8b3","Type":"ContainerStarted","Data":"858d823a658af9ceb5010f0e63df4a55f2731b523bfc28408c568b2e08a89db5"}
Jul 23 11:36:27.361279 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:27.361228    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-prdbw" podStartSLOduration=8.051536553 podStartE2EDuration="41.361212731s" totalImagesPullingTime="33.309676178s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 11:35:46 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:36:27.359860004 +0000 UTC m=+41.752797791" watchObservedRunningTime="2026-07-23 11:36:27.361212731 +0000 UTC m=+41.754150519"
Jul 23 11:36:27.544354 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:27.544239    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:36:27.744494 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:27.744458    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:36:28.143905 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:28.143854    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:36:28.244537 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:28.244494    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:36:28.345339 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:28.345292    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:36:28.383422 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:28.383388    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/94d3e57f-8c9b-48c6-9566-8998f808f363-registry-tls\") pod \"image-registry-668768fb6b-zlz7l\" (UID: \"94d3e57f-8c9b-48c6-9566-8998f808f363\") " pod="openshift-image-registry/image-registry-668768fb6b-zlz7l"
Jul 23 11:36:28.383554 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:28.383442    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/867645c0-1161-4522-b68f-f597f275ed37-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-r4f2b\" (UID: \"867645c0-1161-4522-b68f-f597f275ed37\") " pod="openshift-insights/insights-runtime-extractor-r4f2b"
Jul 23 11:36:28.383554 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:28.383477    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/804c9466-58e2-4986-b0bf-2e85cba3d0e4-metrics-tls\") pod \"dns-default-t2l85\" (UID: \"804c9466-58e2-4986-b0bf-2e85cba3d0e4\") " pod="openshift-dns/dns-default-t2l85"
Jul 23 11:36:28.383645 ip-10-0-141-229 kubenswrapper[2563]: E0723 11:36:28.383595    2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 23 11:36:28.383678 ip-10-0-141-229 kubenswrapper[2563]: E0723 11:36:28.383670    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/867645c0-1161-4522-b68f-f597f275ed37-insights-runtime-extractor-tls podName:867645c0-1161-4522-b68f-f597f275ed37 nodeName:}" failed. No retries permitted until 2026-07-23 11:36:36.383655092 +0000 UTC m=+50.776592880 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/867645c0-1161-4522-b68f-f597f275ed37-insights-runtime-extractor-tls") pod "insights-runtime-extractor-r4f2b" (UID: "867645c0-1161-4522-b68f-f597f275ed37") : secret "insights-runtime-extractor-tls" not found
Jul 23 11:36:28.386728 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:28.386705    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/804c9466-58e2-4986-b0bf-2e85cba3d0e4-metrics-tls\") pod \"dns-default-t2l85\" (UID: \"804c9466-58e2-4986-b0bf-2e85cba3d0e4\") " pod="openshift-dns/dns-default-t2l85"
Jul 23 11:36:28.386830 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:28.386808    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/94d3e57f-8c9b-48c6-9566-8998f808f363-registry-tls\") pod \"image-registry-668768fb6b-zlz7l\" (UID: \"94d3e57f-8c9b-48c6-9566-8998f808f363\") " pod="openshift-image-registry/image-registry-668768fb6b-zlz7l"
Jul 23 11:36:28.415520 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:28.415436    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-t2l85"
Jul 23 11:36:28.551335 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:28.551302    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-t2l85"]
Jul 23 11:36:28.554222 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:36:28.554198    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod804c9466_58e2_4986_b0bf_2e85cba3d0e4.slice/crio-bb76c3cf97dca5b4a8cacdec486a6df33d0d07c7de26df0c8aa8f894db41d7ea WatchSource:0}: Error finding container bb76c3cf97dca5b4a8cacdec486a6df33d0d07c7de26df0c8aa8f894db41d7ea: Status 404 returned error can't find the container with id bb76c3cf97dca5b4a8cacdec486a6df33d0d07c7de26df0c8aa8f894db41d7ea
Jul 23 11:36:28.660510 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:28.660465    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-668768fb6b-zlz7l"
Jul 23 11:36:28.767027 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:28.766960    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:36:28.836115 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:28.836085    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-668768fb6b-zlz7l"]
Jul 23 11:36:28.838984 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:36:28.838958    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94d3e57f_8c9b_48c6_9566_8998f808f363.slice/crio-b61815cdabf7a5e938f540b62646b492194be97f54824d2ac638053591a996ef WatchSource:0}: Error finding container b61815cdabf7a5e938f540b62646b492194be97f54824d2ac638053591a996ef: Status 404 returned error can't find the container with id b61815cdabf7a5e938f540b62646b492194be97f54824d2ac638053591a996ef
Jul 23 11:36:29.344646 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:29.344558    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-668768fb6b-zlz7l" event={"ID":"94d3e57f-8c9b-48c6-9566-8998f808f363","Type":"ContainerStarted","Data":"8dc813ad7730c3d1de7bbbcb396f8ea32048440ae7ab0dba7f2d333ef2f0e141"}
Jul 23 11:36:29.344646 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:29.344626    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-668768fb6b-zlz7l" event={"ID":"94d3e57f-8c9b-48c6-9566-8998f808f363","Type":"ContainerStarted","Data":"b61815cdabf7a5e938f540b62646b492194be97f54824d2ac638053591a996ef"}
Jul 23 11:36:29.345376 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:29.345354    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-668768fb6b-zlz7l"
Jul 23 11:36:29.346798 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:29.346772    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-t2l85" event={"ID":"804c9466-58e2-4986-b0bf-2e85cba3d0e4","Type":"ContainerStarted","Data":"bb76c3cf97dca5b4a8cacdec486a6df33d0d07c7de26df0c8aa8f894db41d7ea"}
Jul 23 11:36:29.356641 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:29.356610    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:36:29.370409 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:29.370077    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-668768fb6b-zlz7l" podStartSLOduration=32.370059641 podStartE2EDuration="32.370059641s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 11:35:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:36:29.36947595 +0000 UTC m=+43.762413738" watchObservedRunningTime="2026-07-23 11:36:29.370059641 +0000 UTC m=+43.762997420"
Jul 23 11:36:29.489416 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:29.489380    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:36:29.587210 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:29.587168    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:36:29.950043 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:29.950000    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:36:30.149259 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:30.149219    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:36:30.759558 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:30.759514    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:36:31.178052 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:31.178017    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:36:31.353560 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:31.353463    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-t2l85" event={"ID":"804c9466-58e2-4986-b0bf-2e85cba3d0e4","Type":"ContainerStarted","Data":"d08e2bb3277e6127889e2deb6c1f197d074b932b95d9bdeb64dcde34eccc1605"}
Jul 23 11:36:31.353560 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:31.353528    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-t2l85" event={"ID":"804c9466-58e2-4986-b0bf-2e85cba3d0e4","Type":"ContainerStarted","Data":"3b01b36a9b300c0606ff79a1cd11c540e3ba5ee77c8da2179166ccd3c4bb55cc"}
Jul 23 11:36:31.375672 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:31.375623    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-t2l85" podStartSLOduration=9.10658578 podStartE2EDuration="11.375605479s" totalImagesPullingTime="2.269019699s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 11:36:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:36:31.374128552 +0000 UTC m=+45.767066339" watchObservedRunningTime="2026-07-23 11:36:31.375605479 +0000 UTC m=+45.768543267"
Jul 23 11:36:31.561003 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:31.560967    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:36:31.957983 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:31.957944    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:36:32.059529 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:32.059489    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:36:32.356217 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:32.356138    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-t2l85"
Jul 23 11:36:32.413245 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:32.413206    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:36:32.559418 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:32.559384    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:36:32.957442 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:32.957403    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:36:33.151599 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:33.151565    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:36:33.831015 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:33.830967    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:36:35.626068 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:35.626025    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:36:35.807824 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:35.807785    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:36:35.915132 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:35.915041    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:36:36.016443 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:36.016406    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:36:36.301824 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:36.301722    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:36:36.374180 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:36.374145    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:36:36.449563 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:36.449524    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/867645c0-1161-4522-b68f-f597f275ed37-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-r4f2b\" (UID: \"867645c0-1161-4522-b68f-f597f275ed37\") " pod="openshift-insights/insights-runtime-extractor-r4f2b"
Jul 23 11:36:36.452029 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:36.452003    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/867645c0-1161-4522-b68f-f597f275ed37-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-r4f2b\" (UID: \"867645c0-1161-4522-b68f-f597f275ed37\") " pod="openshift-insights/insights-runtime-extractor-r4f2b"
Jul 23 11:36:36.494142 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:36.494108    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-r4f2b"
Jul 23 11:36:36.496363 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:36.496330    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:36:36.559836 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:36.559749    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:36:36.644611 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:36.644581    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-r4f2b"]
Jul 23 11:36:36.648106 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:36:36.648074    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod867645c0_1161_4522_b68f_f597f275ed37.slice/crio-d1af666fb1e303a0ab591aaf655887427f8ef8259624935a6d012ce1fbf1b6f9 WatchSource:0}: Error finding container d1af666fb1e303a0ab591aaf655887427f8ef8259624935a6d012ce1fbf1b6f9: Status 404 returned error can't find the container with id d1af666fb1e303a0ab591aaf655887427f8ef8259624935a6d012ce1fbf1b6f9
Jul 23 11:36:36.719814 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:36.719765    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:36:36.819341 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:36.819247    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:36:37.306300 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:37.306243    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:36:37.373137 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:37.373099    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-r4f2b" event={"ID":"867645c0-1161-4522-b68f-f597f275ed37","Type":"ContainerStarted","Data":"d1af666fb1e303a0ab591aaf655887427f8ef8259624935a6d012ce1fbf1b6f9"}
Jul 23 11:36:37.529522 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:37.529486    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:36:37.781931 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:37.781890    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:36:38.201095 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:38.201055    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:36:38.302905 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:38.302864    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:36:38.573633 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:38.573545    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:36:38.686189 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:38.686149    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:36:40.384125 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:40.384095    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-7bc7489cb5-b7zjr"]
Jul 23 11:36:40.480182 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:40.480150    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:36:40.527567 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:40.527525    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-668768fb6b-zlz7l"]
Jul 23 11:36:40.527783 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:40.527589    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-7bc7489cb5-b7zjr"]
Jul 23 11:36:40.527783 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:40.527619    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-5bc67dcd46-nfpg5"]
Jul 23 11:36:40.527906 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:40.527789    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-b7zjr"
Jul 23 11:36:40.531832 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:40.531666    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Jul 23 11:36:40.531832 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:40.531726    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Jul 23 11:36:40.531832 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:40.531726    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-t5stm\""
Jul 23 11:36:40.532819 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:40.532801    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-c78dd7fdc-z2h9v"]
Jul 23 11:36:40.532954 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:40.532936    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-5bc67dcd46-nfpg5"
Jul 23 11:36:40.535194 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:40.535177    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-5bc67dcd46-nfpg5"]
Jul 23 11:36:40.535289 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:40.535205    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-c78dd7fdc-z2h9v"]
Jul 23 11:36:40.535289 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:40.535277    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-c78dd7fdc-z2h9v"
Jul 23 11:36:40.535980 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:40.535962    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 23 11:36:40.536655 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:40.536513    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-bn7rj\""
Jul 23 11:36:40.536655 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:40.536542    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 23 11:36:40.579073 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:40.579039    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:36:40.586048 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:40.585988    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/3320cd4a-2c2f-49fe-99fd-acd7be999bf1-nginx-conf\") pod \"networking-console-plugin-7bc7489cb5-b7zjr\" (UID: \"3320cd4a-2c2f-49fe-99fd-acd7be999bf1\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-b7zjr"
Jul 23 11:36:40.586184 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:40.586067    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9faf18f6-bd93-4f36-999b-ddb434ba06be-installation-pull-secrets\") pod \"image-registry-c78dd7fdc-z2h9v\" (UID: \"9faf18f6-bd93-4f36-999b-ddb434ba06be\") " pod="openshift-image-registry/image-registry-c78dd7fdc-z2h9v"
Jul 23 11:36:40.586184 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:40.586114    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-blq8x\" (UniqueName: \"kubernetes.io/projected/9faf18f6-bd93-4f36-999b-ddb434ba06be-kube-api-access-blq8x\") pod \"image-registry-c78dd7fdc-z2h9v\" (UID: \"9faf18f6-bd93-4f36-999b-ddb434ba06be\") " pod="openshift-image-registry/image-registry-c78dd7fdc-z2h9v"
Jul 23 11:36:40.586184 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:40.586154    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qs5mq\" (UniqueName: \"kubernetes.io/projected/2ab262b2-8386-4891-86d0-1620a18357c6-kube-api-access-qs5mq\") pod \"downloads-5bc67dcd46-nfpg5\" (UID: \"2ab262b2-8386-4891-86d0-1620a18357c6\") " pod="openshift-console/downloads-5bc67dcd46-nfpg5"
Jul 23 11:36:40.586184 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:40.586178    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9faf18f6-bd93-4f36-999b-ddb434ba06be-trusted-ca\") pod \"image-registry-c78dd7fdc-z2h9v\" (UID: \"9faf18f6-bd93-4f36-999b-ddb434ba06be\") " pod="openshift-image-registry/image-registry-c78dd7fdc-z2h9v"
Jul 23 11:36:40.586349 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:40.586233    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9faf18f6-bd93-4f36-999b-ddb434ba06be-image-registry-private-configuration\") pod \"image-registry-c78dd7fdc-z2h9v\" (UID: \"9faf18f6-bd93-4f36-999b-ddb434ba06be\") " pod="openshift-image-registry/image-registry-c78dd7fdc-z2h9v"
Jul 23 11:36:40.586349 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:40.586257    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3320cd4a-2c2f-49fe-99fd-acd7be999bf1-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-b7zjr\" (UID: \"3320cd4a-2c2f-49fe-99fd-acd7be999bf1\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-b7zjr"
Jul 23 11:36:40.586349 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:40.586315    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9faf18f6-bd93-4f36-999b-ddb434ba06be-registry-tls\") pod \"image-registry-c78dd7fdc-z2h9v\" (UID: \"9faf18f6-bd93-4f36-999b-ddb434ba06be\") " pod="openshift-image-registry/image-registry-c78dd7fdc-z2h9v"
Jul 23 11:36:40.586450 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:40.586349    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9faf18f6-bd93-4f36-999b-ddb434ba06be-bound-sa-token\") pod \"image-registry-c78dd7fdc-z2h9v\" (UID: \"9faf18f6-bd93-4f36-999b-ddb434ba06be\") " pod="openshift-image-registry/image-registry-c78dd7fdc-z2h9v"
Jul 23 11:36:40.586450 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:40.586381    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9faf18f6-bd93-4f36-999b-ddb434ba06be-registry-certificates\") pod \"image-registry-c78dd7fdc-z2h9v\" (UID: \"9faf18f6-bd93-4f36-999b-ddb434ba06be\") " pod="openshift-image-registry/image-registry-c78dd7fdc-z2h9v"
Jul 23 11:36:40.586450 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:40.586409    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9faf18f6-bd93-4f36-999b-ddb434ba06be-ca-trust-extracted\") pod \"image-registry-c78dd7fdc-z2h9v\" (UID: \"9faf18f6-bd93-4f36-999b-ddb434ba06be\") " pod="openshift-image-registry/image-registry-c78dd7fdc-z2h9v"
Jul 23 11:36:40.672252 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:40.672155    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:36:40.687251 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:40.687215    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9faf18f6-bd93-4f36-999b-ddb434ba06be-registry-tls\") pod \"image-registry-c78dd7fdc-z2h9v\" (UID: \"9faf18f6-bd93-4f36-999b-ddb434ba06be\") " pod="openshift-image-registry/image-registry-c78dd7fdc-z2h9v"
Jul 23 11:36:40.687437 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:40.687259    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9faf18f6-bd93-4f36-999b-ddb434ba06be-bound-sa-token\") pod \"image-registry-c78dd7fdc-z2h9v\" (UID: \"9faf18f6-bd93-4f36-999b-ddb434ba06be\") " pod="openshift-image-registry/image-registry-c78dd7fdc-z2h9v"
Jul 23 11:36:40.687437 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:40.687289    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9faf18f6-bd93-4f36-999b-ddb434ba06be-registry-certificates\") pod \"image-registry-c78dd7fdc-z2h9v\" (UID: \"9faf18f6-bd93-4f36-999b-ddb434ba06be\") " pod="openshift-image-registry/image-registry-c78dd7fdc-z2h9v"
Jul 23 11:36:40.687437 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:40.687315    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9faf18f6-bd93-4f36-999b-ddb434ba06be-ca-trust-extracted\") pod \"image-registry-c78dd7fdc-z2h9v\" (UID: \"9faf18f6-bd93-4f36-999b-ddb434ba06be\") " pod="openshift-image-registry/image-registry-c78dd7fdc-z2h9v"
Jul 23 11:36:40.687437 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:40.687346    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/3320cd4a-2c2f-49fe-99fd-acd7be999bf1-nginx-conf\") pod \"networking-console-plugin-7bc7489cb5-b7zjr\" (UID: \"3320cd4a-2c2f-49fe-99fd-acd7be999bf1\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-b7zjr"
Jul 23 11:36:40.687437 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:40.687369    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9faf18f6-bd93-4f36-999b-ddb434ba06be-installation-pull-secrets\") pod \"image-registry-c78dd7fdc-z2h9v\" (UID: \"9faf18f6-bd93-4f36-999b-ddb434ba06be\") " pod="openshift-image-registry/image-registry-c78dd7fdc-z2h9v"
Jul 23 11:36:40.687437 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:40.687400    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-blq8x\" (UniqueName: \"kubernetes.io/projected/9faf18f6-bd93-4f36-999b-ddb434ba06be-kube-api-access-blq8x\") pod \"image-registry-c78dd7fdc-z2h9v\" (UID: \"9faf18f6-bd93-4f36-999b-ddb434ba06be\") " pod="openshift-image-registry/image-registry-c78dd7fdc-z2h9v"
Jul 23 11:36:40.687765 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:40.687444    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qs5mq\" (UniqueName: \"kubernetes.io/projected/2ab262b2-8386-4891-86d0-1620a18357c6-kube-api-access-qs5mq\") pod \"downloads-5bc67dcd46-nfpg5\" (UID: \"2ab262b2-8386-4891-86d0-1620a18357c6\") " pod="openshift-console/downloads-5bc67dcd46-nfpg5"
Jul 23 11:36:40.687765 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:40.687479    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9faf18f6-bd93-4f36-999b-ddb434ba06be-trusted-ca\") pod \"image-registry-c78dd7fdc-z2h9v\" (UID: \"9faf18f6-bd93-4f36-999b-ddb434ba06be\") " pod="openshift-image-registry/image-registry-c78dd7fdc-z2h9v"
Jul 23 11:36:40.687765 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:40.687532    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9faf18f6-bd93-4f36-999b-ddb434ba06be-image-registry-private-configuration\") pod \"image-registry-c78dd7fdc-z2h9v\" (UID: \"9faf18f6-bd93-4f36-999b-ddb434ba06be\") " pod="openshift-image-registry/image-registry-c78dd7fdc-z2h9v"
Jul 23 11:36:40.687765 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:40.687563    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3320cd4a-2c2f-49fe-99fd-acd7be999bf1-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-b7zjr\" (UID: \"3320cd4a-2c2f-49fe-99fd-acd7be999bf1\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-b7zjr"
Jul 23 11:36:40.688246 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:40.688126    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/3320cd4a-2c2f-49fe-99fd-acd7be999bf1-nginx-conf\") pod \"networking-console-plugin-7bc7489cb5-b7zjr\" (UID: \"3320cd4a-2c2f-49fe-99fd-acd7be999bf1\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-b7zjr"
Jul 23 11:36:40.688529 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:40.688366    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9faf18f6-bd93-4f36-999b-ddb434ba06be-ca-trust-extracted\") pod \"image-registry-c78dd7fdc-z2h9v\" (UID: \"9faf18f6-bd93-4f36-999b-ddb434ba06be\") " pod="openshift-image-registry/image-registry-c78dd7fdc-z2h9v"
Jul 23 11:36:40.688599 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:40.688530    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9faf18f6-bd93-4f36-999b-ddb434ba06be-registry-certificates\") pod \"image-registry-c78dd7fdc-z2h9v\" (UID: \"9faf18f6-bd93-4f36-999b-ddb434ba06be\") " pod="openshift-image-registry/image-registry-c78dd7fdc-z2h9v"
Jul 23 11:36:40.689232 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:40.689205    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9faf18f6-bd93-4f36-999b-ddb434ba06be-trusted-ca\") pod \"image-registry-c78dd7fdc-z2h9v\" (UID: \"9faf18f6-bd93-4f36-999b-ddb434ba06be\") " pod="openshift-image-registry/image-registry-c78dd7fdc-z2h9v"
Jul 23 11:36:40.690726 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:40.690701    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3320cd4a-2c2f-49fe-99fd-acd7be999bf1-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-b7zjr\" (UID: \"3320cd4a-2c2f-49fe-99fd-acd7be999bf1\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-b7zjr"
Jul 23 11:36:40.690871 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:40.690825    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9faf18f6-bd93-4f36-999b-ddb434ba06be-installation-pull-secrets\") pod \"image-registry-c78dd7fdc-z2h9v\" (UID: \"9faf18f6-bd93-4f36-999b-ddb434ba06be\") " pod="openshift-image-registry/image-registry-c78dd7fdc-z2h9v"
Jul 23 11:36:40.690871 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:40.690836    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9faf18f6-bd93-4f36-999b-ddb434ba06be-registry-tls\") pod \"image-registry-c78dd7fdc-z2h9v\" (UID: \"9faf18f6-bd93-4f36-999b-ddb434ba06be\") " pod="openshift-image-registry/image-registry-c78dd7fdc-z2h9v"
Jul 23 11:36:40.691259 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:40.691234    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9faf18f6-bd93-4f36-999b-ddb434ba06be-image-registry-private-configuration\") pod \"image-registry-c78dd7fdc-z2h9v\" (UID: \"9faf18f6-bd93-4f36-999b-ddb434ba06be\") " pod="openshift-image-registry/image-registry-c78dd7fdc-z2h9v"
Jul 23 11:36:40.696864 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:40.696821    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9faf18f6-bd93-4f36-999b-ddb434ba06be-bound-sa-token\") pod \"image-registry-c78dd7fdc-z2h9v\" (UID: \"9faf18f6-bd93-4f36-999b-ddb434ba06be\") " pod="openshift-image-registry/image-registry-c78dd7fdc-z2h9v"
Jul 23 11:36:40.697781 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:40.697149    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qs5mq\" (UniqueName: \"kubernetes.io/projected/2ab262b2-8386-4891-86d0-1620a18357c6-kube-api-access-qs5mq\") pod \"downloads-5bc67dcd46-nfpg5\" (UID: \"2ab262b2-8386-4891-86d0-1620a18357c6\") " pod="openshift-console/downloads-5bc67dcd46-nfpg5"
Jul 23 11:36:40.699192 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:40.699173    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-blq8x\" (UniqueName: \"kubernetes.io/projected/9faf18f6-bd93-4f36-999b-ddb434ba06be-kube-api-access-blq8x\") pod \"image-registry-c78dd7fdc-z2h9v\" (UID: \"9faf18f6-bd93-4f36-999b-ddb434ba06be\") " pod="openshift-image-registry/image-registry-c78dd7fdc-z2h9v"
Jul 23 11:36:40.770545 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:40.770504    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:36:40.841885 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:40.841843    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-b7zjr"
Jul 23 11:36:40.848983 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:40.848956    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-5bc67dcd46-nfpg5"
Jul 23 11:36:40.854956 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:40.854934    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-c78dd7fdc-z2h9v"
Jul 23 11:36:40.880119 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:40.880082    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:36:41.531157 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:41.531108    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:36:41.632377 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:41.632300    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:36:41.703416 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:41.703390    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-c78dd7fdc-z2h9v"]
Jul 23 11:36:41.706172 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:36:41.706146    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9faf18f6_bd93_4f36_999b_ddb434ba06be.slice/crio-b572bb600dc10a3b6d05f472257c02bd9ce157b798443b7d02504a506ebea795 WatchSource:0}: Error finding container b572bb600dc10a3b6d05f472257c02bd9ce157b798443b7d02504a506ebea795: Status 404 returned error can't find the container with id b572bb600dc10a3b6d05f472257c02bd9ce157b798443b7d02504a506ebea795
Jul 23 11:36:41.941195 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:41.941157    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-7bc7489cb5-b7zjr"]
Jul 23 11:36:41.945117 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:36:41.945092    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3320cd4a_2c2f_49fe_99fd_acd7be999bf1.slice/crio-f5bde461be2d0a7c41c50038c7af95680e4c12e8d0c9c6672e08a67c1d3b77cd WatchSource:0}: Error finding container f5bde461be2d0a7c41c50038c7af95680e4c12e8d0c9c6672e08a67c1d3b77cd: Status 404 returned error can't find the container with id f5bde461be2d0a7c41c50038c7af95680e4c12e8d0c9c6672e08a67c1d3b77cd
Jul 23 11:36:41.945117 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:41.945102    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-5bc67dcd46-nfpg5"]
Jul 23 11:36:41.947169 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:36:41.947120    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ab262b2_8386_4891_86d0_1620a18357c6.slice/crio-83b92fffa76718973f3f380ffefc1f583a3a9a30cf7295e8adac3cccaea9f9fc WatchSource:0}: Error finding container 83b92fffa76718973f3f380ffefc1f583a3a9a30cf7295e8adac3cccaea9f9fc: Status 404 returned error can't find the container with id 83b92fffa76718973f3f380ffefc1f583a3a9a30cf7295e8adac3cccaea9f9fc
Jul 23 11:36:42.360987 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:42.360901    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-t2l85"
Jul 23 11:36:42.388561 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:42.388521    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-b7zjr" event={"ID":"3320cd4a-2c2f-49fe-99fd-acd7be999bf1","Type":"ContainerStarted","Data":"f5bde461be2d0a7c41c50038c7af95680e4c12e8d0c9c6672e08a67c1d3b77cd"}
Jul 23 11:36:42.390778 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:42.390724    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-5bc67dcd46-nfpg5" event={"ID":"2ab262b2-8386-4891-86d0-1620a18357c6","Type":"ContainerStarted","Data":"83b92fffa76718973f3f380ffefc1f583a3a9a30cf7295e8adac3cccaea9f9fc"}
Jul 23 11:36:42.392780 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:42.392713    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-r4f2b" event={"ID":"867645c0-1161-4522-b68f-f597f275ed37","Type":"ContainerStarted","Data":"a7209fc7f644b818c9be30680a30c7d09e69e01e81457203f6cda8cd0c43d2d3"}
Jul 23 11:36:42.396118 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:42.396043    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-c78dd7fdc-z2h9v" event={"ID":"9faf18f6-bd93-4f36-999b-ddb434ba06be","Type":"ContainerStarted","Data":"f70ed023af4295204e5f92379300eec3099c14a613c4a9e56e01d347d587a10c"}
Jul 23 11:36:42.396118 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:42.396089    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-c78dd7fdc-z2h9v" event={"ID":"9faf18f6-bd93-4f36-999b-ddb434ba06be","Type":"ContainerStarted","Data":"b572bb600dc10a3b6d05f472257c02bd9ce157b798443b7d02504a506ebea795"}
Jul 23 11:36:42.396292 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:42.396220    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-c78dd7fdc-z2h9v"
Jul 23 11:36:46.321472 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:46.320338    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-nhm9p"
Jul 23 11:36:46.353657 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:46.353592    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-c78dd7fdc-z2h9v" podStartSLOduration=6.353576584 podStartE2EDuration="6.353576584s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 11:36:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:36:42.418575406 +0000 UTC m=+56.811513193" watchObservedRunningTime="2026-07-23 11:36:46.353576584 +0000 UTC m=+60.746514373"
Jul 23 11:36:46.415058 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:46.415015    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-b7zjr" event={"ID":"3320cd4a-2c2f-49fe-99fd-acd7be999bf1","Type":"ContainerStarted","Data":"412c43ca149045f6633650160563ef54163951cc6feb29e0e19f6c51922a5b47"}
Jul 23 11:36:46.417709 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:46.417612    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-r4f2b" event={"ID":"867645c0-1161-4522-b68f-f597f275ed37","Type":"ContainerStarted","Data":"9440fe46ab36a8af014fdbb1bc0ee2c1f5e37a281670a4c52b390e8492d35450"}
Jul 23 11:36:46.488262 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:46.488228    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:36:46.587935 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:46.587779    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:36:46.690666 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:46.690593    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:36:46.772777 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:46.772730    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:36:48.476190 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:48.476151    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:36:48.575716 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:48.575667    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:36:48.678096 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:48.678056    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:36:48.889651 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:48.889557    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:36:48.963744 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:48.963668    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:36:49.024900 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:49.024864    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:36:49.114882 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:49.114847    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:36:49.215033 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:49.214961    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:36:50.433799 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:50.433760    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-r4f2b" event={"ID":"867645c0-1161-4522-b68f-f597f275ed37","Type":"ContainerStarted","Data":"65c5d329427b50579afc0b383bb33b6c47cb092e0a3a09447b0409acbccd5c28"}
Jul 23 11:36:50.457345 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:50.457281    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-r4f2b" podStartSLOduration=17.568549628 podStartE2EDuration="30.457258819s" totalImagesPullingTime="12.888709191s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 11:36:20 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:36:50.456598945 +0000 UTC m=+64.849536734" watchObservedRunningTime="2026-07-23 11:36:50.457258819 +0000 UTC m=+64.850196610"
Jul 23 11:36:50.457540 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:50.457417    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-b7zjr" podStartSLOduration=6.815780655 podStartE2EDuration="10.457406642s" totalImagesPullingTime="3.641625987s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 11:36:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:36:46.462834665 +0000 UTC m=+60.855772471" watchObservedRunningTime="2026-07-23 11:36:50.457406642 +0000 UTC m=+64.850344430"
Jul 23 11:36:50.533770 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:50.533741    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-668768fb6b-zlz7l"
Jul 23 11:36:50.879271 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:50.879237    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cbdff1cb-47fd-4ed9-bec9-419ea8da3aaa-metrics-certs\") pod \"network-metrics-daemon-pbmtq\" (UID: \"cbdff1cb-47fd-4ed9-bec9-419ea8da3aaa\") " pod="openshift-multus/network-metrics-daemon-pbmtq"
Jul 23 11:36:50.879467 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:50.879318    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-528jd\" (UniqueName: \"kubernetes.io/projected/b30ea3bc-3059-4d68-934c-32cde3d06fbb-kube-api-access-528jd\") pod \"network-check-target-l6jwv\" (UID: \"b30ea3bc-3059-4d68-934c-32cde3d06fbb\") " pod="openshift-network-diagnostics/network-check-target-l6jwv"
Jul 23 11:36:50.879467 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:50.879366    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7a0946cf-0f04-49d5-ab74-6f64a48f8e5d-original-pull-secret\") pod \"global-pull-secret-syncer-cvbbt\" (UID: \"7a0946cf-0f04-49d5-ab74-6f64a48f8e5d\") " pod="kube-system/global-pull-secret-syncer-cvbbt"
Jul 23 11:36:50.882108 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:50.882086    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 23 11:36:50.882238 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:50.882116    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 23 11:36:50.882287 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:50.882262    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 23 11:36:50.893338 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:50.893315    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 23 11:36:50.893499 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:50.893478    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7a0946cf-0f04-49d5-ab74-6f64a48f8e5d-original-pull-secret\") pod \"global-pull-secret-syncer-cvbbt\" (UID: \"7a0946cf-0f04-49d5-ab74-6f64a48f8e5d\") " pod="kube-system/global-pull-secret-syncer-cvbbt"
Jul 23 11:36:50.893557 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:50.893528    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cbdff1cb-47fd-4ed9-bec9-419ea8da3aaa-metrics-certs\") pod \"network-metrics-daemon-pbmtq\" (UID: \"cbdff1cb-47fd-4ed9-bec9-419ea8da3aaa\") " pod="openshift-multus/network-metrics-daemon-pbmtq"
Jul 23 11:36:50.904259 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:50.904233    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-528jd\" (UniqueName: \"kubernetes.io/projected/b30ea3bc-3059-4d68-934c-32cde3d06fbb-kube-api-access-528jd\") pod \"network-check-target-l6jwv\" (UID: \"b30ea3bc-3059-4d68-934c-32cde3d06fbb\") " pod="openshift-network-diagnostics/network-check-target-l6jwv"
Jul 23 11:36:51.086848 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:51.086803    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-7flkj\""
Jul 23 11:36:51.093429 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:51.093399    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-xtz4b\""
Jul 23 11:36:51.095213 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:51.095189    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pbmtq"
Jul 23 11:36:51.101411 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:51.101388    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l6jwv"
Jul 23 11:36:51.183298 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:51.183224    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cvbbt"
Jul 23 11:36:51.256334 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:51.256305    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-pbmtq"]
Jul 23 11:36:51.262198 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:36:51.262166    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbdff1cb_47fd_4ed9_bec9_419ea8da3aaa.slice/crio-b96a30adefb1260b2fb2c0c935d59640acee272a63f82036d53b2908d0baebba WatchSource:0}: Error finding container b96a30adefb1260b2fb2c0c935d59640acee272a63f82036d53b2908d0baebba: Status 404 returned error can't find the container with id b96a30adefb1260b2fb2c0c935d59640acee272a63f82036d53b2908d0baebba
Jul 23 11:36:51.280909 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:51.280879    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-l6jwv"]
Jul 23 11:36:51.341015 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:51.340980    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-cvbbt"]
Jul 23 11:36:51.344371 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:36:51.344336    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a0946cf_0f04_49d5_ab74_6f64a48f8e5d.slice/crio-fde796ed45461d09286831feb1827d1f10cdaf607cdc17c312aca95b37a47100 WatchSource:0}: Error finding container fde796ed45461d09286831feb1827d1f10cdaf607cdc17c312aca95b37a47100: Status 404 returned error can't find the container with id fde796ed45461d09286831feb1827d1f10cdaf607cdc17c312aca95b37a47100
Jul 23 11:36:51.439030 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:51.438932    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-cvbbt" event={"ID":"7a0946cf-0f04-49d5-ab74-6f64a48f8e5d","Type":"ContainerStarted","Data":"fde796ed45461d09286831feb1827d1f10cdaf607cdc17c312aca95b37a47100"}
Jul 23 11:36:51.440235 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:51.440201    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-l6jwv" event={"ID":"b30ea3bc-3059-4d68-934c-32cde3d06fbb","Type":"ContainerStarted","Data":"5fcd25cc2f54100a0dc64bb648f601cc7997b9101a06741d92d01347e351b194"}
Jul 23 11:36:51.441462 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:51.441437    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-pbmtq" event={"ID":"cbdff1cb-47fd-4ed9-bec9-419ea8da3aaa","Type":"ContainerStarted","Data":"b96a30adefb1260b2fb2c0c935d59640acee272a63f82036d53b2908d0baebba"}
Jul 23 11:36:51.500431 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:51.500396    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:36:51.570977 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:51.570942    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:36:51.649915 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:51.649877    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:36:51.716291 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:51.716188    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:36:52.872267 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:52.872232    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:36:52.972664 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:52.972604    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:36:53.138914 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:53.138676    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:36:53.211945 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:53.211886    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:36:53.281523 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:53.281474    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:36:54.717298 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:54.717263    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:36:54.806412 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:54.806382    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:36:54.884432 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:54.884398    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:36:55.570484 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:55.570377    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:36:55.662379 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:55.662351    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:36:58.182612 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:36:58.182563    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:37:00.366578 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:00.366527    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:37:00.465061 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:00.465011    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:37:00.731946 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:00.731805    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:37:00.842645 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:00.842614    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:37:00.956758 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:00.956726    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:37:01.060910 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:01.060810    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:37:01.165943 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:01.165889    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:37:01.261032 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:01.260998    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:37:01.329221 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:01.329131    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-kmbvx"]
Jul 23 11:37:01.337453 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:01.337419    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-kmbvx"
Jul 23 11:37:01.340586 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:01.340551    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 23 11:37:01.340874 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:01.340857    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 23 11:37:01.343201 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:01.343171    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-mhh7m\""
Jul 23 11:37:01.343555 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:01.343528    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 23 11:37:01.343661 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:01.343618    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 23 11:37:01.343661 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:01.343642    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 23 11:37:01.343975 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:01.343956    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 23 11:37:01.393358 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:01.393325    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:37:01.464643 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:01.464597    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d79b19eb-59df-4f8c-affe-c67eb5e142aa-node-exporter-wtmp\") pod \"node-exporter-kmbvx\" (UID: \"d79b19eb-59df-4f8c-affe-c67eb5e142aa\") " pod="openshift-monitoring/node-exporter-kmbvx"
Jul 23 11:37:01.464643 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:01.464651    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d79b19eb-59df-4f8c-affe-c67eb5e142aa-node-exporter-tls\") pod \"node-exporter-kmbvx\" (UID: \"d79b19eb-59df-4f8c-affe-c67eb5e142aa\") " pod="openshift-monitoring/node-exporter-kmbvx"
Jul 23 11:37:01.464898 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:01.464674    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d79b19eb-59df-4f8c-affe-c67eb5e142aa-sys\") pod \"node-exporter-kmbvx\" (UID: \"d79b19eb-59df-4f8c-affe-c67eb5e142aa\") " pod="openshift-monitoring/node-exporter-kmbvx"
Jul 23 11:37:01.464898 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:01.464772    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d79b19eb-59df-4f8c-affe-c67eb5e142aa-node-exporter-accelerators-collector-config\") pod \"node-exporter-kmbvx\" (UID: \"d79b19eb-59df-4f8c-affe-c67eb5e142aa\") " pod="openshift-monitoring/node-exporter-kmbvx"
Jul 23 11:37:01.464898 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:01.464817    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75jdq\" (UniqueName: \"kubernetes.io/projected/d79b19eb-59df-4f8c-affe-c67eb5e142aa-kube-api-access-75jdq\") pod \"node-exporter-kmbvx\" (UID: \"d79b19eb-59df-4f8c-affe-c67eb5e142aa\") " pod="openshift-monitoring/node-exporter-kmbvx"
Jul 23 11:37:01.464898 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:01.464847    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d79b19eb-59df-4f8c-affe-c67eb5e142aa-node-exporter-textfile\") pod \"node-exporter-kmbvx\" (UID: \"d79b19eb-59df-4f8c-affe-c67eb5e142aa\") " pod="openshift-monitoring/node-exporter-kmbvx"
Jul 23 11:37:01.464898 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:01.464890    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d79b19eb-59df-4f8c-affe-c67eb5e142aa-metrics-client-ca\") pod \"node-exporter-kmbvx\" (UID: \"d79b19eb-59df-4f8c-affe-c67eb5e142aa\") " pod="openshift-monitoring/node-exporter-kmbvx"
Jul 23 11:37:01.465106 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:01.464986    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d79b19eb-59df-4f8c-affe-c67eb5e142aa-root\") pod \"node-exporter-kmbvx\" (UID: \"d79b19eb-59df-4f8c-affe-c67eb5e142aa\") " pod="openshift-monitoring/node-exporter-kmbvx"
Jul 23 11:37:01.465106 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:01.465033    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d79b19eb-59df-4f8c-affe-c67eb5e142aa-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-kmbvx\" (UID: \"d79b19eb-59df-4f8c-affe-c67eb5e142aa\") " pod="openshift-monitoring/node-exporter-kmbvx"
Jul 23 11:37:01.566484 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:01.566417    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d79b19eb-59df-4f8c-affe-c67eb5e142aa-node-exporter-tls\") pod \"node-exporter-kmbvx\" (UID: \"d79b19eb-59df-4f8c-affe-c67eb5e142aa\") " pod="openshift-monitoring/node-exporter-kmbvx"
Jul 23 11:37:01.566484 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:01.566476    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d79b19eb-59df-4f8c-affe-c67eb5e142aa-sys\") pod \"node-exporter-kmbvx\" (UID: \"d79b19eb-59df-4f8c-affe-c67eb5e142aa\") " pod="openshift-monitoring/node-exporter-kmbvx"
Jul 23 11:37:01.566770 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:01.566521    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d79b19eb-59df-4f8c-affe-c67eb5e142aa-node-exporter-accelerators-collector-config\") pod \"node-exporter-kmbvx\" (UID: \"d79b19eb-59df-4f8c-affe-c67eb5e142aa\") " pod="openshift-monitoring/node-exporter-kmbvx"
Jul 23 11:37:01.566770 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:01.566557    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75jdq\" (UniqueName: \"kubernetes.io/projected/d79b19eb-59df-4f8c-affe-c67eb5e142aa-kube-api-access-75jdq\") pod \"node-exporter-kmbvx\" (UID: \"d79b19eb-59df-4f8c-affe-c67eb5e142aa\") " pod="openshift-monitoring/node-exporter-kmbvx"
Jul 23 11:37:01.566770 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:01.566570    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d79b19eb-59df-4f8c-affe-c67eb5e142aa-sys\") pod \"node-exporter-kmbvx\" (UID: \"d79b19eb-59df-4f8c-affe-c67eb5e142aa\") " pod="openshift-monitoring/node-exporter-kmbvx"
Jul 23 11:37:01.566770 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:01.566587    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d79b19eb-59df-4f8c-affe-c67eb5e142aa-node-exporter-textfile\") pod \"node-exporter-kmbvx\" (UID: \"d79b19eb-59df-4f8c-affe-c67eb5e142aa\") " pod="openshift-monitoring/node-exporter-kmbvx"
Jul 23 11:37:01.566770 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:01.566611    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d79b19eb-59df-4f8c-affe-c67eb5e142aa-metrics-client-ca\") pod \"node-exporter-kmbvx\" (UID: \"d79b19eb-59df-4f8c-affe-c67eb5e142aa\") " pod="openshift-monitoring/node-exporter-kmbvx"
Jul 23 11:37:01.566770 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:01.566636    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d79b19eb-59df-4f8c-affe-c67eb5e142aa-root\") pod \"node-exporter-kmbvx\" (UID: \"d79b19eb-59df-4f8c-affe-c67eb5e142aa\") " pod="openshift-monitoring/node-exporter-kmbvx"
Jul 23 11:37:01.566770 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:01.566716    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d79b19eb-59df-4f8c-affe-c67eb5e142aa-root\") pod \"node-exporter-kmbvx\" (UID: \"d79b19eb-59df-4f8c-affe-c67eb5e142aa\") " pod="openshift-monitoring/node-exporter-kmbvx"
Jul 23 11:37:01.567096 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:01.566771    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d79b19eb-59df-4f8c-affe-c67eb5e142aa-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-kmbvx\" (UID: \"d79b19eb-59df-4f8c-affe-c67eb5e142aa\") " pod="openshift-monitoring/node-exporter-kmbvx"
Jul 23 11:37:01.567096 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:01.566811    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d79b19eb-59df-4f8c-affe-c67eb5e142aa-node-exporter-wtmp\") pod \"node-exporter-kmbvx\" (UID: \"d79b19eb-59df-4f8c-affe-c67eb5e142aa\") " pod="openshift-monitoring/node-exporter-kmbvx"
Jul 23 11:37:01.567096 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:01.566973    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d79b19eb-59df-4f8c-affe-c67eb5e142aa-node-exporter-wtmp\") pod \"node-exporter-kmbvx\" (UID: \"d79b19eb-59df-4f8c-affe-c67eb5e142aa\") " pod="openshift-monitoring/node-exporter-kmbvx"
Jul 23 11:37:01.567096 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:01.567076    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d79b19eb-59df-4f8c-affe-c67eb5e142aa-node-exporter-textfile\") pod \"node-exporter-kmbvx\" (UID: \"d79b19eb-59df-4f8c-affe-c67eb5e142aa\") " pod="openshift-monitoring/node-exporter-kmbvx"
Jul 23 11:37:01.567301 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:01.567248    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d79b19eb-59df-4f8c-affe-c67eb5e142aa-node-exporter-accelerators-collector-config\") pod \"node-exporter-kmbvx\" (UID: \"d79b19eb-59df-4f8c-affe-c67eb5e142aa\") " pod="openshift-monitoring/node-exporter-kmbvx"
Jul 23 11:37:01.567355 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:01.567328    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d79b19eb-59df-4f8c-affe-c67eb5e142aa-metrics-client-ca\") pod \"node-exporter-kmbvx\" (UID: \"d79b19eb-59df-4f8c-affe-c67eb5e142aa\") " pod="openshift-monitoring/node-exporter-kmbvx"
Jul 23 11:37:01.569278 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:01.569255    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d79b19eb-59df-4f8c-affe-c67eb5e142aa-node-exporter-tls\") pod \"node-exporter-kmbvx\" (UID: \"d79b19eb-59df-4f8c-affe-c67eb5e142aa\") " pod="openshift-monitoring/node-exporter-kmbvx"
Jul 23 11:37:01.569744 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:01.569708    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d79b19eb-59df-4f8c-affe-c67eb5e142aa-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-kmbvx\" (UID: \"d79b19eb-59df-4f8c-affe-c67eb5e142aa\") " pod="openshift-monitoring/node-exporter-kmbvx"
Jul 23 11:37:01.584793 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:01.584701    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-75jdq\" (UniqueName: \"kubernetes.io/projected/d79b19eb-59df-4f8c-affe-c67eb5e142aa-kube-api-access-75jdq\") pod \"node-exporter-kmbvx\" (UID: \"d79b19eb-59df-4f8c-affe-c67eb5e142aa\") " pod="openshift-monitoring/node-exporter-kmbvx"
Jul 23 11:37:01.587120 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:01.587083    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:37:01.650085 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:01.650050    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-kmbvx"
Jul 23 11:37:01.757005 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:01.756955    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:37:01.865856 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:01.865797    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:37:02.074506 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:02.074455    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:37:02.178510 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:02.178422    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:37:02.291383 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:02.291333    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:37:02.480488 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:02.480399    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:37:03.404224 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:03.404192    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-c78dd7fdc-z2h9v"
Jul 23 11:37:03.564328 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:37:03.564284    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd79b19eb_59df_4f8c_affe_c67eb5e142aa.slice/crio-6db56ca8529dec4f5fa37cae1cbcaf2ee3df926320fe50e0829acc0a46011dc8 WatchSource:0}: Error finding container 6db56ca8529dec4f5fa37cae1cbcaf2ee3df926320fe50e0829acc0a46011dc8: Status 404 returned error can't find the container with id 6db56ca8529dec4f5fa37cae1cbcaf2ee3df926320fe50e0829acc0a46011dc8
Jul 23 11:37:03.586510 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:03.586477    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:37:03.687943 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:03.687847    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:37:03.969874 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:03.969833    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:37:04.048918 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:04.048809    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:37:04.259909 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:04.259821    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:37:04.298995 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:04.298947    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-7dfbd954f7-jjdbk"]
Jul 23 11:37:04.319317 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:04.319271    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-7dfbd954f7-jjdbk"
Jul 23 11:37:04.322372 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:04.322338    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Jul 23 11:37:04.322819 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:04.322797    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Jul 23 11:37:04.323597 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:04.323369    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Jul 23 11:37:04.323597 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:04.323420    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Jul 23 11:37:04.323597 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:04.323369    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-4sbbso9jmdfv1\""
Jul 23 11:37:04.330042 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:04.324080    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-lxjqm\""
Jul 23 11:37:04.330042 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:04.324186    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-7dfbd954f7-jjdbk"]
Jul 23 11:37:04.330462 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:04.330431    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Jul 23 11:37:04.351527 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:04.351488    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:37:04.487675 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:04.487627    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0b575b60-75a2-4539-bc75-08e2c621bc0b-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7dfbd954f7-jjdbk\" (UID: \"0b575b60-75a2-4539-bc75-08e2c621bc0b\") " pod="openshift-monitoring/thanos-querier-7dfbd954f7-jjdbk"
Jul 23 11:37:04.487875 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:04.487707    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/0b575b60-75a2-4539-bc75-08e2c621bc0b-secret-thanos-querier-tls\") pod \"thanos-querier-7dfbd954f7-jjdbk\" (UID: \"0b575b60-75a2-4539-bc75-08e2c621bc0b\") " pod="openshift-monitoring/thanos-querier-7dfbd954f7-jjdbk"
Jul 23 11:37:04.487875 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:04.487769    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0b575b60-75a2-4539-bc75-08e2c621bc0b-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7dfbd954f7-jjdbk\" (UID: \"0b575b60-75a2-4539-bc75-08e2c621bc0b\") " pod="openshift-monitoring/thanos-querier-7dfbd954f7-jjdbk"
Jul 23 11:37:04.487875 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:04.487803    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7xgt\" (UniqueName: \"kubernetes.io/projected/0b575b60-75a2-4539-bc75-08e2c621bc0b-kube-api-access-l7xgt\") pod \"thanos-querier-7dfbd954f7-jjdbk\" (UID: \"0b575b60-75a2-4539-bc75-08e2c621bc0b\") " pod="openshift-monitoring/thanos-querier-7dfbd954f7-jjdbk"
Jul 23 11:37:04.487875 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:04.487851    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0b575b60-75a2-4539-bc75-08e2c621bc0b-secret-grpc-tls\") pod \"thanos-querier-7dfbd954f7-jjdbk\" (UID: \"0b575b60-75a2-4539-bc75-08e2c621bc0b\") " pod="openshift-monitoring/thanos-querier-7dfbd954f7-jjdbk"
Jul 23 11:37:04.488076 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:04.487883    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/0b575b60-75a2-4539-bc75-08e2c621bc0b-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7dfbd954f7-jjdbk\" (UID: \"0b575b60-75a2-4539-bc75-08e2c621bc0b\") " pod="openshift-monitoring/thanos-querier-7dfbd954f7-jjdbk"
Jul 23 11:37:04.488076 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:04.487917    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/0b575b60-75a2-4539-bc75-08e2c621bc0b-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7dfbd954f7-jjdbk\" (UID: \"0b575b60-75a2-4539-bc75-08e2c621bc0b\") " pod="openshift-monitoring/thanos-querier-7dfbd954f7-jjdbk"
Jul 23 11:37:04.488076 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:04.487974    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0b575b60-75a2-4539-bc75-08e2c621bc0b-metrics-client-ca\") pod \"thanos-querier-7dfbd954f7-jjdbk\" (UID: \"0b575b60-75a2-4539-bc75-08e2c621bc0b\") " pod="openshift-monitoring/thanos-querier-7dfbd954f7-jjdbk"
Jul 23 11:37:04.494901 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:04.494856    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-cvbbt" event={"ID":"7a0946cf-0f04-49d5-ab74-6f64a48f8e5d","Type":"ContainerStarted","Data":"4019a10588f68b7679ae8f969115951ce4eea2bce8366e1799f227c11db12f96"}
Jul 23 11:37:04.497432 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:04.497357    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-l6jwv" event={"ID":"b30ea3bc-3059-4d68-934c-32cde3d06fbb","Type":"ContainerStarted","Data":"85121446b56868589563b09e19345b3a90126eb82834ec12d614017d0d97b51a"}
Jul 23 11:37:04.497621 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:04.497472    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-l6jwv"
Jul 23 11:37:04.500609 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:04.500563    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-pbmtq" event={"ID":"cbdff1cb-47fd-4ed9-bec9-419ea8da3aaa","Type":"ContainerStarted","Data":"278761bc1558bf457dbede5072cfbf10abb016bbe6611ac7678f02147d40b85d"}
Jul 23 11:37:04.500828 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:04.500634    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-pbmtq" event={"ID":"cbdff1cb-47fd-4ed9-bec9-419ea8da3aaa","Type":"ContainerStarted","Data":"95b0770e4de5e020844895ef1f7033600aeedab4299ef75d3842ea5b4b8b0a30"}
Jul 23 11:37:04.502262 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:04.502228    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-kmbvx" event={"ID":"d79b19eb-59df-4f8c-affe-c67eb5e142aa","Type":"ContainerStarted","Data":"6db56ca8529dec4f5fa37cae1cbcaf2ee3df926320fe50e0829acc0a46011dc8"}
Jul 23 11:37:04.504631 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:04.504590    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-5bc67dcd46-nfpg5" event={"ID":"2ab262b2-8386-4891-86d0-1620a18357c6","Type":"ContainerStarted","Data":"1a3e1b1c94db32649ec84b163739997907aacb1c5b8c4adb2614727be0083ec9"}
Jul 23 11:37:04.505580 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:04.505557    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-5bc67dcd46-nfpg5"
Jul 23 11:37:04.513786 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:04.513656    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-cvbbt" podStartSLOduration=66.299220762 podStartE2EDuration="1m18.513633835s" totalImagesPullingTime="12.214413073s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 11:35:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:37:04.512716041 +0000 UTC m=+78.905653830" watchObservedRunningTime="2026-07-23 11:37:04.513633835 +0000 UTC m=+78.906571623"
Jul 23 11:37:04.517035 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:04.516988    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-5bc67dcd46-nfpg5"
Jul 23 11:37:04.535120 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:04.535062    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-l6jwv" podStartSLOduration=66.254939695 podStartE2EDuration="1m18.535043371s" totalImagesPullingTime="12.280103676s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 11:35:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:37:04.533077417 +0000 UTC m=+78.926015226" watchObservedRunningTime="2026-07-23 11:37:04.535043371 +0000 UTC m=+78.927981151"
Jul 23 11:37:04.555336 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:04.555270    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-5bc67dcd46-nfpg5" podStartSLOduration=2.926732736 podStartE2EDuration="24.555250806s" totalImagesPullingTime="21.62851807s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 11:36:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:37:04.554013394 +0000 UTC m=+78.946951194" watchObservedRunningTime="2026-07-23 11:37:04.555250806 +0000 UTC m=+78.948188597"
Jul 23 11:37:04.580398 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:04.580358    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:37:04.588985 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:04.588933    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0b575b60-75a2-4539-bc75-08e2c621bc0b-metrics-client-ca\") pod \"thanos-querier-7dfbd954f7-jjdbk\" (UID: \"0b575b60-75a2-4539-bc75-08e2c621bc0b\") " pod="openshift-monitoring/thanos-querier-7dfbd954f7-jjdbk"
Jul 23 11:37:04.589189 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:04.589040    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0b575b60-75a2-4539-bc75-08e2c621bc0b-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7dfbd954f7-jjdbk\" (UID: \"0b575b60-75a2-4539-bc75-08e2c621bc0b\") " pod="openshift-monitoring/thanos-querier-7dfbd954f7-jjdbk"
Jul 23 11:37:04.589189 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:04.589072    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/0b575b60-75a2-4539-bc75-08e2c621bc0b-secret-thanos-querier-tls\") pod \"thanos-querier-7dfbd954f7-jjdbk\" (UID: \"0b575b60-75a2-4539-bc75-08e2c621bc0b\") " pod="openshift-monitoring/thanos-querier-7dfbd954f7-jjdbk"
Jul 23 11:37:04.589189 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:04.589143    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0b575b60-75a2-4539-bc75-08e2c621bc0b-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7dfbd954f7-jjdbk\" (UID: \"0b575b60-75a2-4539-bc75-08e2c621bc0b\") " pod="openshift-monitoring/thanos-querier-7dfbd954f7-jjdbk"
Jul 23 11:37:04.589189 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:04.589169    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7xgt\" (UniqueName: \"kubernetes.io/projected/0b575b60-75a2-4539-bc75-08e2c621bc0b-kube-api-access-l7xgt\") pod \"thanos-querier-7dfbd954f7-jjdbk\" (UID: \"0b575b60-75a2-4539-bc75-08e2c621bc0b\") " pod="openshift-monitoring/thanos-querier-7dfbd954f7-jjdbk"
Jul 23 11:37:04.589397 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:04.589222    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0b575b60-75a2-4539-bc75-08e2c621bc0b-secret-grpc-tls\") pod \"thanos-querier-7dfbd954f7-jjdbk\" (UID: \"0b575b60-75a2-4539-bc75-08e2c621bc0b\") " pod="openshift-monitoring/thanos-querier-7dfbd954f7-jjdbk"
Jul 23 11:37:04.589397 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:04.589269    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/0b575b60-75a2-4539-bc75-08e2c621bc0b-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7dfbd954f7-jjdbk\" (UID: \"0b575b60-75a2-4539-bc75-08e2c621bc0b\") " pod="openshift-monitoring/thanos-querier-7dfbd954f7-jjdbk"
Jul 23 11:37:04.589397 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:04.589300    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/0b575b60-75a2-4539-bc75-08e2c621bc0b-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7dfbd954f7-jjdbk\" (UID: \"0b575b60-75a2-4539-bc75-08e2c621bc0b\") " pod="openshift-monitoring/thanos-querier-7dfbd954f7-jjdbk"
Jul 23 11:37:04.590504 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:04.589854    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0b575b60-75a2-4539-bc75-08e2c621bc0b-metrics-client-ca\") pod \"thanos-querier-7dfbd954f7-jjdbk\" (UID: \"0b575b60-75a2-4539-bc75-08e2c621bc0b\") " pod="openshift-monitoring/thanos-querier-7dfbd954f7-jjdbk"
Jul 23 11:37:04.592643 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:04.592576    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/0b575b60-75a2-4539-bc75-08e2c621bc0b-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7dfbd954f7-jjdbk\" (UID: \"0b575b60-75a2-4539-bc75-08e2c621bc0b\") " pod="openshift-monitoring/thanos-querier-7dfbd954f7-jjdbk"
Jul 23 11:37:04.594186 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:04.593608    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0b575b60-75a2-4539-bc75-08e2c621bc0b-secret-grpc-tls\") pod \"thanos-querier-7dfbd954f7-jjdbk\" (UID: \"0b575b60-75a2-4539-bc75-08e2c621bc0b\") " pod="openshift-monitoring/thanos-querier-7dfbd954f7-jjdbk"
Jul 23 11:37:04.594186 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:04.594135    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0b575b60-75a2-4539-bc75-08e2c621bc0b-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7dfbd954f7-jjdbk\" (UID: \"0b575b60-75a2-4539-bc75-08e2c621bc0b\") " pod="openshift-monitoring/thanos-querier-7dfbd954f7-jjdbk"
Jul 23 11:37:04.594831 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:04.594777    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0b575b60-75a2-4539-bc75-08e2c621bc0b-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7dfbd954f7-jjdbk\" (UID: \"0b575b60-75a2-4539-bc75-08e2c621bc0b\") " pod="openshift-monitoring/thanos-querier-7dfbd954f7-jjdbk"
Jul 23 11:37:04.594954 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:04.594863    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/0b575b60-75a2-4539-bc75-08e2c621bc0b-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7dfbd954f7-jjdbk\" (UID: \"0b575b60-75a2-4539-bc75-08e2c621bc0b\") " pod="openshift-monitoring/thanos-querier-7dfbd954f7-jjdbk"
Jul 23 11:37:04.595318 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:04.595292    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/0b575b60-75a2-4539-bc75-08e2c621bc0b-secret-thanos-querier-tls\") pod \"thanos-querier-7dfbd954f7-jjdbk\" (UID: \"0b575b60-75a2-4539-bc75-08e2c621bc0b\") " pod="openshift-monitoring/thanos-querier-7dfbd954f7-jjdbk"
Jul 23 11:37:04.603889 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:04.603727    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7xgt\" (UniqueName: \"kubernetes.io/projected/0b575b60-75a2-4539-bc75-08e2c621bc0b-kube-api-access-l7xgt\") pod \"thanos-querier-7dfbd954f7-jjdbk\" (UID: \"0b575b60-75a2-4539-bc75-08e2c621bc0b\") " pod="openshift-monitoring/thanos-querier-7dfbd954f7-jjdbk"
Jul 23 11:37:04.605487 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:04.605436    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-pbmtq" podStartSLOduration=66.310577242 podStartE2EDuration="1m18.605413754s" totalImagesPullingTime="12.294836512s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 11:35:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:37:04.579618009 +0000 UTC m=+78.972555799" watchObservedRunningTime="2026-07-23 11:37:04.605413754 +0000 UTC m=+78.998351543"
Jul 23 11:37:04.636798 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:04.636753    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-7dfbd954f7-jjdbk"
Jul 23 11:37:04.677860 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:04.677824    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:37:04.782566 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:04.782322    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:37:04.872185 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:04.872147    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-7dfbd954f7-jjdbk"]
Jul 23 11:37:04.876388 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:04.876292    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:37:04.964411 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:37:04.964356    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b575b60_75a2_4539_bc75_08e2c621bc0b.slice/crio-e7daa3255874322e887c3e4cf9b8fe0b8dbff36b998daa3e2fbd4c0aeece12ff WatchSource:0}: Error finding container e7daa3255874322e887c3e4cf9b8fe0b8dbff36b998daa3e2fbd4c0aeece12ff: Status 404 returned error can't find the container with id e7daa3255874322e887c3e4cf9b8fe0b8dbff36b998daa3e2fbd4c0aeece12ff
Jul 23 11:37:05.512168 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:05.512129    2563 generic.go:356] "Generic (PLEG): container finished" podID="d79b19eb-59df-4f8c-affe-c67eb5e142aa" containerID="9a8f4e2e067aa56b7bde9ac126fc3835980b2b21131c9fd9527c4a200059ecd1" exitCode=0
Jul 23 11:37:05.512361 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:05.512233    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-kmbvx" event={"ID":"d79b19eb-59df-4f8c-affe-c67eb5e142aa","Type":"ContainerDied","Data":"9a8f4e2e067aa56b7bde9ac126fc3835980b2b21131c9fd9527c4a200059ecd1"}
Jul 23 11:37:05.514219 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:05.514188    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7dfbd954f7-jjdbk" event={"ID":"0b575b60-75a2-4539-bc75-08e2c621bc0b","Type":"ContainerStarted","Data":"e7daa3255874322e887c3e4cf9b8fe0b8dbff36b998daa3e2fbd4c0aeece12ff"}
Jul 23 11:37:05.554149 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:05.554105    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-668768fb6b-zlz7l" podUID="94d3e57f-8c9b-48c6-9566-8998f808f363" containerName="registry" containerID="cri-o://8dc813ad7730c3d1de7bbbcb396f8ea32048440ae7ab0dba7f2d333ef2f0e141" gracePeriod=30
Jul 23 11:37:05.849350 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:05.849312    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-8564b6f9f6-9wwdl"]
Jul 23 11:37:05.872712 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:05.872632    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-8564b6f9f6-9wwdl"
Jul 23 11:37:05.878514 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:05.878477    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-8564b6f9f6-9wwdl"]
Jul 23 11:37:05.882793 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:05.882160    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-668768fb6b-zlz7l"
Jul 23 11:37:05.886599 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:05.885466    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 23 11:37:05.886599 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:05.885960    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Jul 23 11:37:05.886599 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:05.886236    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-hxv4p\""
Jul 23 11:37:05.886599 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:05.886462    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-c0k34f8d22mus\""
Jul 23 11:37:05.887532 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:05.886712    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Jul 23 11:37:05.887532 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:05.886938    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Jul 23 11:37:05.992465 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:05.992424    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:37:05.999176 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:05.999123    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/94d3e57f-8c9b-48c6-9566-8998f808f363-installation-pull-secrets\") pod \"94d3e57f-8c9b-48c6-9566-8998f808f363\" (UID: \"94d3e57f-8c9b-48c6-9566-8998f808f363\") "
Jul 23 11:37:05.999365 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:05.999187    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/94d3e57f-8c9b-48c6-9566-8998f808f363-registry-tls\") pod \"94d3e57f-8c9b-48c6-9566-8998f808f363\" (UID: \"94d3e57f-8c9b-48c6-9566-8998f808f363\") "
Jul 23 11:37:05.999365 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:05.999227    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/94d3e57f-8c9b-48c6-9566-8998f808f363-trusted-ca\") pod \"94d3e57f-8c9b-48c6-9566-8998f808f363\" (UID: \"94d3e57f-8c9b-48c6-9566-8998f808f363\") "
Jul 23 11:37:05.999365 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:05.999262    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/94d3e57f-8c9b-48c6-9566-8998f808f363-registry-certificates\") pod \"94d3e57f-8c9b-48c6-9566-8998f808f363\" (UID: \"94d3e57f-8c9b-48c6-9566-8998f808f363\") "
Jul 23 11:37:05.999365 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:05.999290    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/94d3e57f-8c9b-48c6-9566-8998f808f363-bound-sa-token\") pod \"94d3e57f-8c9b-48c6-9566-8998f808f363\" (UID: \"94d3e57f-8c9b-48c6-9566-8998f808f363\") "
Jul 23 11:37:05.999365 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:05.999330    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mmwqd\" (UniqueName: \"kubernetes.io/projected/94d3e57f-8c9b-48c6-9566-8998f808f363-kube-api-access-mmwqd\") pod \"94d3e57f-8c9b-48c6-9566-8998f808f363\" (UID: \"94d3e57f-8c9b-48c6-9566-8998f808f363\") "
Jul 23 11:37:05.999617 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:05.999384    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/94d3e57f-8c9b-48c6-9566-8998f808f363-ca-trust-extracted\") pod \"94d3e57f-8c9b-48c6-9566-8998f808f363\" (UID: \"94d3e57f-8c9b-48c6-9566-8998f808f363\") "
Jul 23 11:37:05.999617 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:05.999420    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/94d3e57f-8c9b-48c6-9566-8998f808f363-image-registry-private-configuration\") pod \"94d3e57f-8c9b-48c6-9566-8998f808f363\" (UID: \"94d3e57f-8c9b-48c6-9566-8998f808f363\") "
Jul 23 11:37:05.999617 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:05.999599    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a3b3aca3-c21e-4f24-be22-4c626e03bfbf-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-8564b6f9f6-9wwdl\" (UID: \"a3b3aca3-c21e-4f24-be22-4c626e03bfbf\") " pod="openshift-monitoring/metrics-server-8564b6f9f6-9wwdl"
Jul 23 11:37:05.999814 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:05.999638    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/a3b3aca3-c21e-4f24-be22-4c626e03bfbf-client-ca-bundle\") pod \"metrics-server-8564b6f9f6-9wwdl\" (UID: \"a3b3aca3-c21e-4f24-be22-4c626e03bfbf\") " pod="openshift-monitoring/metrics-server-8564b6f9f6-9wwdl"
Jul 23 11:37:05.999814 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:05.999672    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjdgw\" (UniqueName: \"kubernetes.io/projected/a3b3aca3-c21e-4f24-be22-4c626e03bfbf-kube-api-access-gjdgw\") pod \"metrics-server-8564b6f9f6-9wwdl\" (UID: \"a3b3aca3-c21e-4f24-be22-4c626e03bfbf\") " pod="openshift-monitoring/metrics-server-8564b6f9f6-9wwdl"
Jul 23 11:37:05.999814 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:05.999756    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/a3b3aca3-c21e-4f24-be22-4c626e03bfbf-secret-metrics-server-tls\") pod \"metrics-server-8564b6f9f6-9wwdl\" (UID: \"a3b3aca3-c21e-4f24-be22-4c626e03bfbf\") " pod="openshift-monitoring/metrics-server-8564b6f9f6-9wwdl"
Jul 23 11:37:05.999814 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:05.999779    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/a3b3aca3-c21e-4f24-be22-4c626e03bfbf-audit-log\") pod \"metrics-server-8564b6f9f6-9wwdl\" (UID: \"a3b3aca3-c21e-4f24-be22-4c626e03bfbf\") " pod="openshift-monitoring/metrics-server-8564b6f9f6-9wwdl"
Jul 23 11:37:05.999814 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:05.999804    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/a3b3aca3-c21e-4f24-be22-4c626e03bfbf-metrics-server-audit-profiles\") pod \"metrics-server-8564b6f9f6-9wwdl\" (UID: \"a3b3aca3-c21e-4f24-be22-4c626e03bfbf\") " pod="openshift-monitoring/metrics-server-8564b6f9f6-9wwdl"
Jul 23 11:37:06.000053 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:05.999837    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/a3b3aca3-c21e-4f24-be22-4c626e03bfbf-secret-metrics-server-client-certs\") pod \"metrics-server-8564b6f9f6-9wwdl\" (UID: \"a3b3aca3-c21e-4f24-be22-4c626e03bfbf\") " pod="openshift-monitoring/metrics-server-8564b6f9f6-9wwdl"
Jul 23 11:37:06.000053 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:05.999830    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94d3e57f-8c9b-48c6-9566-8998f808f363-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "94d3e57f-8c9b-48c6-9566-8998f808f363" (UID: "94d3e57f-8c9b-48c6-9566-8998f808f363"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 11:37:06.001566 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:06.001505    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94d3e57f-8c9b-48c6-9566-8998f808f363-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "94d3e57f-8c9b-48c6-9566-8998f808f363" (UID: "94d3e57f-8c9b-48c6-9566-8998f808f363"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 11:37:06.005954 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:06.005503    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94d3e57f-8c9b-48c6-9566-8998f808f363-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "94d3e57f-8c9b-48c6-9566-8998f808f363" (UID: "94d3e57f-8c9b-48c6-9566-8998f808f363"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 11:37:06.006861 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:06.006777    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94d3e57f-8c9b-48c6-9566-8998f808f363-kube-api-access-mmwqd" (OuterVolumeSpecName: "kube-api-access-mmwqd") pod "94d3e57f-8c9b-48c6-9566-8998f808f363" (UID: "94d3e57f-8c9b-48c6-9566-8998f808f363"). InnerVolumeSpecName "kube-api-access-mmwqd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 11:37:06.007215 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:06.007153    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/94d3e57f-8c9b-48c6-9566-8998f808f363-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "94d3e57f-8c9b-48c6-9566-8998f808f363" (UID: "94d3e57f-8c9b-48c6-9566-8998f808f363"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 11:37:06.007215 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:06.007199    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94d3e57f-8c9b-48c6-9566-8998f808f363-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "94d3e57f-8c9b-48c6-9566-8998f808f363" (UID: "94d3e57f-8c9b-48c6-9566-8998f808f363"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 11:37:06.007845 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:06.007759    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/94d3e57f-8c9b-48c6-9566-8998f808f363-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "94d3e57f-8c9b-48c6-9566-8998f808f363" (UID: "94d3e57f-8c9b-48c6-9566-8998f808f363"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 11:37:06.016673 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:06.016589    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94d3e57f-8c9b-48c6-9566-8998f808f363-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "94d3e57f-8c9b-48c6-9566-8998f808f363" (UID: "94d3e57f-8c9b-48c6-9566-8998f808f363"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 11:37:06.032332 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:06.032287    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-7b7b7bc8bf-4tn6c"]
Jul 23 11:37:06.032743 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:06.032677    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="94d3e57f-8c9b-48c6-9566-8998f808f363" containerName="registry"
Jul 23 11:37:06.032743 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:06.032717    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="94d3e57f-8c9b-48c6-9566-8998f808f363" containerName="registry"
Jul 23 11:37:06.032966 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:06.032797    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="94d3e57f-8c9b-48c6-9566-8998f808f363" containerName="registry"
Jul 23 11:37:06.068394 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:06.063282    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-7b7b7bc8bf-4tn6c"]
Jul 23 11:37:06.068394 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:06.063411    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-4tn6c"
Jul 23 11:37:06.068394 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:06.066849    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Jul 23 11:37:06.068394 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:06.066876    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-74hs7\""
Jul 23 11:37:06.104346 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:06.104305    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a3b3aca3-c21e-4f24-be22-4c626e03bfbf-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-8564b6f9f6-9wwdl\" (UID: \"a3b3aca3-c21e-4f24-be22-4c626e03bfbf\") " pod="openshift-monitoring/metrics-server-8564b6f9f6-9wwdl"
Jul 23 11:37:06.104542 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:06.104359    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/a3b3aca3-c21e-4f24-be22-4c626e03bfbf-client-ca-bundle\") pod \"metrics-server-8564b6f9f6-9wwdl\" (UID: \"a3b3aca3-c21e-4f24-be22-4c626e03bfbf\") " pod="openshift-monitoring/metrics-server-8564b6f9f6-9wwdl"
Jul 23 11:37:06.104542 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:06.104387    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjdgw\" (UniqueName: \"kubernetes.io/projected/a3b3aca3-c21e-4f24-be22-4c626e03bfbf-kube-api-access-gjdgw\") pod \"metrics-server-8564b6f9f6-9wwdl\" (UID: \"a3b3aca3-c21e-4f24-be22-4c626e03bfbf\") " pod="openshift-monitoring/metrics-server-8564b6f9f6-9wwdl"
Jul 23 11:37:06.104542 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:06.104430    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/a3b3aca3-c21e-4f24-be22-4c626e03bfbf-secret-metrics-server-tls\") pod \"metrics-server-8564b6f9f6-9wwdl\" (UID: \"a3b3aca3-c21e-4f24-be22-4c626e03bfbf\") " pod="openshift-monitoring/metrics-server-8564b6f9f6-9wwdl"
Jul 23 11:37:06.104542 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:06.104453    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/a3b3aca3-c21e-4f24-be22-4c626e03bfbf-audit-log\") pod \"metrics-server-8564b6f9f6-9wwdl\" (UID: \"a3b3aca3-c21e-4f24-be22-4c626e03bfbf\") " pod="openshift-monitoring/metrics-server-8564b6f9f6-9wwdl"
Jul 23 11:37:06.104542 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:06.104474    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/a3b3aca3-c21e-4f24-be22-4c626e03bfbf-metrics-server-audit-profiles\") pod \"metrics-server-8564b6f9f6-9wwdl\" (UID: \"a3b3aca3-c21e-4f24-be22-4c626e03bfbf\") " pod="openshift-monitoring/metrics-server-8564b6f9f6-9wwdl"
Jul 23 11:37:06.104542 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:06.104498    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/a3b3aca3-c21e-4f24-be22-4c626e03bfbf-secret-metrics-server-client-certs\") pod \"metrics-server-8564b6f9f6-9wwdl\" (UID: \"a3b3aca3-c21e-4f24-be22-4c626e03bfbf\") " pod="openshift-monitoring/metrics-server-8564b6f9f6-9wwdl"
Jul 23 11:37:06.104868 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:06.104572    2563 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/94d3e57f-8c9b-48c6-9566-8998f808f363-installation-pull-secrets\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 23 11:37:06.104868 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:06.104589    2563 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/94d3e57f-8c9b-48c6-9566-8998f808f363-registry-tls\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 23 11:37:06.104868 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:06.104604    2563 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/94d3e57f-8c9b-48c6-9566-8998f808f363-trusted-ca\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 23 11:37:06.104868 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:06.104617    2563 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/94d3e57f-8c9b-48c6-9566-8998f808f363-registry-certificates\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 23 11:37:06.104868 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:06.104630    2563 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/94d3e57f-8c9b-48c6-9566-8998f808f363-bound-sa-token\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 23 11:37:06.104868 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:06.104644    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mmwqd\" (UniqueName: \"kubernetes.io/projected/94d3e57f-8c9b-48c6-9566-8998f808f363-kube-api-access-mmwqd\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 23 11:37:06.104868 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:06.104658    2563 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/94d3e57f-8c9b-48c6-9566-8998f808f363-ca-trust-extracted\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 23 11:37:06.104868 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:06.104672    2563 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/94d3e57f-8c9b-48c6-9566-8998f808f363-image-registry-private-configuration\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 23 11:37:06.105249 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:06.105098    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/a3b3aca3-c21e-4f24-be22-4c626e03bfbf-audit-log\") pod \"metrics-server-8564b6f9f6-9wwdl\" (UID: \"a3b3aca3-c21e-4f24-be22-4c626e03bfbf\") " pod="openshift-monitoring/metrics-server-8564b6f9f6-9wwdl"
Jul 23 11:37:06.105309 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:06.105240    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a3b3aca3-c21e-4f24-be22-4c626e03bfbf-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-8564b6f9f6-9wwdl\" (UID: \"a3b3aca3-c21e-4f24-be22-4c626e03bfbf\") " pod="openshift-monitoring/metrics-server-8564b6f9f6-9wwdl"
Jul 23 11:37:06.106092 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:06.106062    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/a3b3aca3-c21e-4f24-be22-4c626e03bfbf-metrics-server-audit-profiles\") pod \"metrics-server-8564b6f9f6-9wwdl\" (UID: \"a3b3aca3-c21e-4f24-be22-4c626e03bfbf\") " pod="openshift-monitoring/metrics-server-8564b6f9f6-9wwdl"
Jul 23 11:37:06.110408 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:06.110379    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/a3b3aca3-c21e-4f24-be22-4c626e03bfbf-secret-metrics-server-client-certs\") pod \"metrics-server-8564b6f9f6-9wwdl\" (UID: \"a3b3aca3-c21e-4f24-be22-4c626e03bfbf\") " pod="openshift-monitoring/metrics-server-8564b6f9f6-9wwdl"
Jul 23 11:37:06.121578 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:06.121372    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:37:06.121990 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:06.121942    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/a3b3aca3-c21e-4f24-be22-4c626e03bfbf-client-ca-bundle\") pod \"metrics-server-8564b6f9f6-9wwdl\" (UID: \"a3b3aca3-c21e-4f24-be22-4c626e03bfbf\") " pod="openshift-monitoring/metrics-server-8564b6f9f6-9wwdl"
Jul 23 11:37:06.123057 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:06.123032    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/a3b3aca3-c21e-4f24-be22-4c626e03bfbf-secret-metrics-server-tls\") pod \"metrics-server-8564b6f9f6-9wwdl\" (UID: \"a3b3aca3-c21e-4f24-be22-4c626e03bfbf\") " pod="openshift-monitoring/metrics-server-8564b6f9f6-9wwdl"
Jul 23 11:37:06.125509 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:06.125480    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjdgw\" (UniqueName: \"kubernetes.io/projected/a3b3aca3-c21e-4f24-be22-4c626e03bfbf-kube-api-access-gjdgw\") pod \"metrics-server-8564b6f9f6-9wwdl\" (UID: \"a3b3aca3-c21e-4f24-be22-4c626e03bfbf\") " pod="openshift-monitoring/metrics-server-8564b6f9f6-9wwdl"
Jul 23 11:37:06.191705 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:06.191570    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-8564b6f9f6-9wwdl"
Jul 23 11:37:06.205306 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:06.205211    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbw6h\" (UniqueName: \"kubernetes.io/projected/af96bf6b-bf0e-4dc4-8a34-71f0955c4bce-kube-api-access-mbw6h\") pod \"monitoring-plugin-7b7b7bc8bf-4tn6c\" (UID: \"af96bf6b-bf0e-4dc4-8a34-71f0955c4bce\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-4tn6c"
Jul 23 11:37:06.205495 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:06.205350    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/af96bf6b-bf0e-4dc4-8a34-71f0955c4bce-monitoring-plugin-cert\") pod \"monitoring-plugin-7b7b7bc8bf-4tn6c\" (UID: \"af96bf6b-bf0e-4dc4-8a34-71f0955c4bce\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-4tn6c"
Jul 23 11:37:06.211809 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:06.211744    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:37:06.306941 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:06.306894    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbw6h\" (UniqueName: \"kubernetes.io/projected/af96bf6b-bf0e-4dc4-8a34-71f0955c4bce-kube-api-access-mbw6h\") pod \"monitoring-plugin-7b7b7bc8bf-4tn6c\" (UID: \"af96bf6b-bf0e-4dc4-8a34-71f0955c4bce\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-4tn6c"
Jul 23 11:37:06.307125 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:06.306983    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/af96bf6b-bf0e-4dc4-8a34-71f0955c4bce-monitoring-plugin-cert\") pod \"monitoring-plugin-7b7b7bc8bf-4tn6c\" (UID: \"af96bf6b-bf0e-4dc4-8a34-71f0955c4bce\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-4tn6c"
Jul 23 11:37:06.310387 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:06.310351    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/af96bf6b-bf0e-4dc4-8a34-71f0955c4bce-monitoring-plugin-cert\") pod \"monitoring-plugin-7b7b7bc8bf-4tn6c\" (UID: \"af96bf6b-bf0e-4dc4-8a34-71f0955c4bce\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-4tn6c"
Jul 23 11:37:06.318461 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:06.318426    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbw6h\" (UniqueName: \"kubernetes.io/projected/af96bf6b-bf0e-4dc4-8a34-71f0955c4bce-kube-api-access-mbw6h\") pod \"monitoring-plugin-7b7b7bc8bf-4tn6c\" (UID: \"af96bf6b-bf0e-4dc4-8a34-71f0955c4bce\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-4tn6c"
Jul 23 11:37:06.396768 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:06.396666    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-4tn6c"
Jul 23 11:37:06.406043 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:06.405983    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-8564b6f9f6-9wwdl"]
Jul 23 11:37:06.531087 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:06.530206    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-kmbvx" event={"ID":"d79b19eb-59df-4f8c-affe-c67eb5e142aa","Type":"ContainerStarted","Data":"fc598410276ffed08cc69c89ad36018215d28803ce475e074271a417a9c7bcce"}
Jul 23 11:37:06.531087 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:06.530277    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-kmbvx" event={"ID":"d79b19eb-59df-4f8c-affe-c67eb5e142aa","Type":"ContainerStarted","Data":"2ae768381d51c42c2629bfd4712a7ceacf908f6c02dbd5a9aad147ba1903f317"}
Jul 23 11:37:06.543314 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:06.543272    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-8564b6f9f6-9wwdl" event={"ID":"a3b3aca3-c21e-4f24-be22-4c626e03bfbf","Type":"ContainerStarted","Data":"99c1403fa235aea6531e9dffaa45969e9670fea620021327f038a5780e13905d"}
Jul 23 11:37:06.545633 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:06.545605    2563 generic.go:356] "Generic (PLEG): container finished" podID="94d3e57f-8c9b-48c6-9566-8998f808f363" containerID="8dc813ad7730c3d1de7bbbcb396f8ea32048440ae7ab0dba7f2d333ef2f0e141" exitCode=0
Jul 23 11:37:06.546322 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:06.546298    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-668768fb6b-zlz7l"
Jul 23 11:37:06.547381 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:06.546882    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-668768fb6b-zlz7l" event={"ID":"94d3e57f-8c9b-48c6-9566-8998f808f363","Type":"ContainerDied","Data":"8dc813ad7730c3d1de7bbbcb396f8ea32048440ae7ab0dba7f2d333ef2f0e141"}
Jul 23 11:37:06.547381 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:06.546939    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-668768fb6b-zlz7l" event={"ID":"94d3e57f-8c9b-48c6-9566-8998f808f363","Type":"ContainerDied","Data":"b61815cdabf7a5e938f540b62646b492194be97f54824d2ac638053591a996ef"}
Jul 23 11:37:06.547381 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:06.546968    2563 scope.go:117] "RemoveContainer" containerID="8dc813ad7730c3d1de7bbbcb396f8ea32048440ae7ab0dba7f2d333ef2f0e141"
Jul 23 11:37:06.579857 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:06.573442    2563 scope.go:117] "RemoveContainer" containerID="8dc813ad7730c3d1de7bbbcb396f8ea32048440ae7ab0dba7f2d333ef2f0e141"
Jul 23 11:37:06.579857 ip-10-0-141-229 kubenswrapper[2563]: E0723 11:37:06.573993    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8dc813ad7730c3d1de7bbbcb396f8ea32048440ae7ab0dba7f2d333ef2f0e141\": container with ID starting with 8dc813ad7730c3d1de7bbbcb396f8ea32048440ae7ab0dba7f2d333ef2f0e141 not found: ID does not exist" containerID="8dc813ad7730c3d1de7bbbcb396f8ea32048440ae7ab0dba7f2d333ef2f0e141"
Jul 23 11:37:06.579857 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:06.574053    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8dc813ad7730c3d1de7bbbcb396f8ea32048440ae7ab0dba7f2d333ef2f0e141"} err="failed to get container status \"8dc813ad7730c3d1de7bbbcb396f8ea32048440ae7ab0dba7f2d333ef2f0e141\": rpc error: code = NotFound desc = could not find container \"8dc813ad7730c3d1de7bbbcb396f8ea32048440ae7ab0dba7f2d333ef2f0e141\": container with ID starting with 8dc813ad7730c3d1de7bbbcb396f8ea32048440ae7ab0dba7f2d333ef2f0e141 not found: ID does not exist"
Jul 23 11:37:06.590925 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:06.587389    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-kmbvx" podStartSLOduration=4.067915102 podStartE2EDuration="5.587367157s" totalImagesPullingTime="1.412263331s" totalInitContainerRuntime="107.188724ms" isStatefulPod=true podCreationTimestamp="2026-07-23 11:37:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:37:06.557848945 +0000 UTC m=+80.950786735" watchObservedRunningTime="2026-07-23 11:37:06.587367157 +0000 UTC m=+80.980304996"
Jul 23 11:37:06.590925 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:06.588080    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-668768fb6b-zlz7l"]
Jul 23 11:37:06.593480 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:06.593448    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-668768fb6b-zlz7l"]
Jul 23 11:37:06.620502 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:06.620471    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-7b7b7bc8bf-4tn6c"]
Jul 23 11:37:06.623899 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:37:06.623858    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf96bf6b_bf0e_4dc4_8a34_71f0955c4bce.slice/crio-fccbfd400a7a2c7b2534d9c8c0f2634d7b03d97efe06b4d6384eeadcb270453f WatchSource:0}: Error finding container fccbfd400a7a2c7b2534d9c8c0f2634d7b03d97efe06b4d6384eeadcb270453f: Status 404 returned error can't find the container with id fccbfd400a7a2c7b2534d9c8c0f2634d7b03d97efe06b4d6384eeadcb270453f
Jul 23 11:37:07.555664 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:07.555626    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-4tn6c" event={"ID":"af96bf6b-bf0e-4dc4-8a34-71f0955c4bce","Type":"ContainerStarted","Data":"fccbfd400a7a2c7b2534d9c8c0f2634d7b03d97efe06b4d6384eeadcb270453f"}
Jul 23 11:37:07.609304 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:07.609267    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:37:07.610250 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:07.610220    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 23 11:37:07.636774 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:07.635435    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 11:37:07.639043 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:07.639008    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 23 11:37:07.639290 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:07.639270    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 23 11:37:07.639520 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:07.639499    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 23 11:37:07.640213 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:07.639840    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 23 11:37:07.640213 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:07.640061    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 23 11:37:07.641250 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:07.641156    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 23 11:37:07.643767 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:07.643414    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 23 11:37:07.652310 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:07.649864    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-pljtt\""
Jul 23 11:37:07.652310 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:07.650149    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 23 11:37:07.652310 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:07.650335    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 23 11:37:07.652310 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:07.650723    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 23 11:37:07.652310 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:07.650940    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 23 11:37:07.652310 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:07.651125    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-f27n8jjki1o2b\""
Jul 23 11:37:07.652310 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:07.651332    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 23 11:37:07.652310 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:07.651502    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 23 11:37:07.652310 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:07.651675    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 23 11:37:07.652310 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:07.652157    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 23 11:37:07.720130 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:07.720089    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/76406e60-8b7f-41cb-ae94-c92bdccc7374-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"76406e60-8b7f-41cb-ae94-c92bdccc7374\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 11:37:07.720317 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:07.720146    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/76406e60-8b7f-41cb-ae94-c92bdccc7374-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"76406e60-8b7f-41cb-ae94-c92bdccc7374\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 11:37:07.720317 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:07.720181    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/76406e60-8b7f-41cb-ae94-c92bdccc7374-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"76406e60-8b7f-41cb-ae94-c92bdccc7374\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 11:37:07.720317 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:07.720225    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/76406e60-8b7f-41cb-ae94-c92bdccc7374-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"76406e60-8b7f-41cb-ae94-c92bdccc7374\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 11:37:07.720317 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:07.720250    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/76406e60-8b7f-41cb-ae94-c92bdccc7374-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"76406e60-8b7f-41cb-ae94-c92bdccc7374\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 11:37:07.720317 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:07.720278    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/76406e60-8b7f-41cb-ae94-c92bdccc7374-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"76406e60-8b7f-41cb-ae94-c92bdccc7374\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 11:37:07.720568 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:07.720320    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/76406e60-8b7f-41cb-ae94-c92bdccc7374-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"76406e60-8b7f-41cb-ae94-c92bdccc7374\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 11:37:07.720568 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:07.720369    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/76406e60-8b7f-41cb-ae94-c92bdccc7374-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"76406e60-8b7f-41cb-ae94-c92bdccc7374\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 11:37:07.720568 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:07.720406    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/76406e60-8b7f-41cb-ae94-c92bdccc7374-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"76406e60-8b7f-41cb-ae94-c92bdccc7374\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 11:37:07.720568 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:07.720434    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/76406e60-8b7f-41cb-ae94-c92bdccc7374-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"76406e60-8b7f-41cb-ae94-c92bdccc7374\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 11:37:07.720568 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:07.720463    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/76406e60-8b7f-41cb-ae94-c92bdccc7374-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"76406e60-8b7f-41cb-ae94-c92bdccc7374\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 11:37:07.720568 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:07.720490    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/76406e60-8b7f-41cb-ae94-c92bdccc7374-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"76406e60-8b7f-41cb-ae94-c92bdccc7374\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 11:37:07.720568 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:07.720524    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/76406e60-8b7f-41cb-ae94-c92bdccc7374-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"76406e60-8b7f-41cb-ae94-c92bdccc7374\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 11:37:07.720568 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:07.720566    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/76406e60-8b7f-41cb-ae94-c92bdccc7374-config\") pod \"prometheus-k8s-0\" (UID: \"76406e60-8b7f-41cb-ae94-c92bdccc7374\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 11:37:07.720991 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:07.720603    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/76406e60-8b7f-41cb-ae94-c92bdccc7374-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"76406e60-8b7f-41cb-ae94-c92bdccc7374\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 11:37:07.720991 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:07.720630    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/76406e60-8b7f-41cb-ae94-c92bdccc7374-config-out\") pod \"prometheus-k8s-0\" (UID: \"76406e60-8b7f-41cb-ae94-c92bdccc7374\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 11:37:07.720991 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:07.720658    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/76406e60-8b7f-41cb-ae94-c92bdccc7374-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"76406e60-8b7f-41cb-ae94-c92bdccc7374\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 11:37:07.720991 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:07.720701    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/76406e60-8b7f-41cb-ae94-c92bdccc7374-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"76406e60-8b7f-41cb-ae94-c92bdccc7374\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 11:37:07.720991 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:07.720727    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/76406e60-8b7f-41cb-ae94-c92bdccc7374-web-config\") pod \"prometheus-k8s-0\" (UID: \"76406e60-8b7f-41cb-ae94-c92bdccc7374\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 11:37:07.720991 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:07.720750    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbzdf\" (UniqueName: \"kubernetes.io/projected/76406e60-8b7f-41cb-ae94-c92bdccc7374-kube-api-access-sbzdf\") pod \"prometheus-k8s-0\" (UID: \"76406e60-8b7f-41cb-ae94-c92bdccc7374\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 11:37:07.821538 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:07.821430    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/76406e60-8b7f-41cb-ae94-c92bdccc7374-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"76406e60-8b7f-41cb-ae94-c92bdccc7374\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 11:37:07.821538 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:07.821480    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/76406e60-8b7f-41cb-ae94-c92bdccc7374-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"76406e60-8b7f-41cb-ae94-c92bdccc7374\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 11:37:07.821538 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:07.821501    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/76406e60-8b7f-41cb-ae94-c92bdccc7374-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"76406e60-8b7f-41cb-ae94-c92bdccc7374\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 11:37:07.821538 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:07.821522    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/76406e60-8b7f-41cb-ae94-c92bdccc7374-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"76406e60-8b7f-41cb-ae94-c92bdccc7374\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 11:37:07.821925 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:07.821556    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/76406e60-8b7f-41cb-ae94-c92bdccc7374-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"76406e60-8b7f-41cb-ae94-c92bdccc7374\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 11:37:07.821925 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:07.821588    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/76406e60-8b7f-41cb-ae94-c92bdccc7374-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"76406e60-8b7f-41cb-ae94-c92bdccc7374\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 11:37:07.821925 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:07.821623    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/76406e60-8b7f-41cb-ae94-c92bdccc7374-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"76406e60-8b7f-41cb-ae94-c92bdccc7374\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 11:37:07.821925 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:07.821657    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/76406e60-8b7f-41cb-ae94-c92bdccc7374-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"76406e60-8b7f-41cb-ae94-c92bdccc7374\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 11:37:07.821925 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:07.821702    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/76406e60-8b7f-41cb-ae94-c92bdccc7374-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"76406e60-8b7f-41cb-ae94-c92bdccc7374\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 11:37:07.821925 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:07.821730    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/76406e60-8b7f-41cb-ae94-c92bdccc7374-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"76406e60-8b7f-41cb-ae94-c92bdccc7374\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 11:37:07.821925 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:07.821760    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/76406e60-8b7f-41cb-ae94-c92bdccc7374-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"76406e60-8b7f-41cb-ae94-c92bdccc7374\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 11:37:07.821925 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:07.821790    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/76406e60-8b7f-41cb-ae94-c92bdccc7374-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"76406e60-8b7f-41cb-ae94-c92bdccc7374\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 11:37:07.821925 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:07.821818    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/76406e60-8b7f-41cb-ae94-c92bdccc7374-config\") pod \"prometheus-k8s-0\" (UID: \"76406e60-8b7f-41cb-ae94-c92bdccc7374\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 11:37:07.821925 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:07.821848    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/76406e60-8b7f-41cb-ae94-c92bdccc7374-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"76406e60-8b7f-41cb-ae94-c92bdccc7374\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 11:37:07.821925 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:07.821876    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/76406e60-8b7f-41cb-ae94-c92bdccc7374-config-out\") pod \"prometheus-k8s-0\" (UID: \"76406e60-8b7f-41cb-ae94-c92bdccc7374\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 11:37:07.821925 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:07.821902    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/76406e60-8b7f-41cb-ae94-c92bdccc7374-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"76406e60-8b7f-41cb-ae94-c92bdccc7374\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 11:37:07.821925 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:07.821927    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/76406e60-8b7f-41cb-ae94-c92bdccc7374-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"76406e60-8b7f-41cb-ae94-c92bdccc7374\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 11:37:07.822517 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:07.821951    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/76406e60-8b7f-41cb-ae94-c92bdccc7374-web-config\") pod \"prometheus-k8s-0\" (UID: \"76406e60-8b7f-41cb-ae94-c92bdccc7374\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 11:37:07.822517 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:07.821975    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbzdf\" (UniqueName: \"kubernetes.io/projected/76406e60-8b7f-41cb-ae94-c92bdccc7374-kube-api-access-sbzdf\") pod \"prometheus-k8s-0\" (UID: \"76406e60-8b7f-41cb-ae94-c92bdccc7374\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 11:37:07.822517 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:07.822017    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/76406e60-8b7f-41cb-ae94-c92bdccc7374-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"76406e60-8b7f-41cb-ae94-c92bdccc7374\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 11:37:07.823955 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:07.823370    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/76406e60-8b7f-41cb-ae94-c92bdccc7374-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"76406e60-8b7f-41cb-ae94-c92bdccc7374\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 11:37:07.829808 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:07.829507    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/76406e60-8b7f-41cb-ae94-c92bdccc7374-web-config\") pod \"prometheus-k8s-0\" (UID: \"76406e60-8b7f-41cb-ae94-c92bdccc7374\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 11:37:07.829808 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:07.829576    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/76406e60-8b7f-41cb-ae94-c92bdccc7374-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"76406e60-8b7f-41cb-ae94-c92bdccc7374\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 11:37:07.831366 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:07.830133    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/76406e60-8b7f-41cb-ae94-c92bdccc7374-config-out\") pod \"prometheus-k8s-0\" (UID: \"76406e60-8b7f-41cb-ae94-c92bdccc7374\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 11:37:07.831366 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:07.830485    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/76406e60-8b7f-41cb-ae94-c92bdccc7374-config\") pod \"prometheus-k8s-0\" (UID: \"76406e60-8b7f-41cb-ae94-c92bdccc7374\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 11:37:07.831366 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:07.830730    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/76406e60-8b7f-41cb-ae94-c92bdccc7374-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"76406e60-8b7f-41cb-ae94-c92bdccc7374\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 11:37:07.831366 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:07.830766    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/76406e60-8b7f-41cb-ae94-c92bdccc7374-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"76406e60-8b7f-41cb-ae94-c92bdccc7374\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 11:37:07.832075 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:07.831836    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/76406e60-8b7f-41cb-ae94-c92bdccc7374-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"76406e60-8b7f-41cb-ae94-c92bdccc7374\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 11:37:07.832392 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:07.832366    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/76406e60-8b7f-41cb-ae94-c92bdccc7374-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"76406e60-8b7f-41cb-ae94-c92bdccc7374\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 11:37:07.832678 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:07.832655    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/76406e60-8b7f-41cb-ae94-c92bdccc7374-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"76406e60-8b7f-41cb-ae94-c92bdccc7374\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 11:37:07.832893 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:07.832869    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/76406e60-8b7f-41cb-ae94-c92bdccc7374-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"76406e60-8b7f-41cb-ae94-c92bdccc7374\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 11:37:07.833386 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:07.833357    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/76406e60-8b7f-41cb-ae94-c92bdccc7374-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"76406e60-8b7f-41cb-ae94-c92bdccc7374\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 11:37:07.834090 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:07.834063    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/76406e60-8b7f-41cb-ae94-c92bdccc7374-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"76406e60-8b7f-41cb-ae94-c92bdccc7374\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 11:37:07.837173 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:07.837143    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/76406e60-8b7f-41cb-ae94-c92bdccc7374-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"76406e60-8b7f-41cb-ae94-c92bdccc7374\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 11:37:07.839847 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:07.839776    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/76406e60-8b7f-41cb-ae94-c92bdccc7374-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"76406e60-8b7f-41cb-ae94-c92bdccc7374\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 11:37:07.840004 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:07.839894    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/76406e60-8b7f-41cb-ae94-c92bdccc7374-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"76406e60-8b7f-41cb-ae94-c92bdccc7374\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 11:37:07.843498 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:07.843418    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/76406e60-8b7f-41cb-ae94-c92bdccc7374-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"76406e60-8b7f-41cb-ae94-c92bdccc7374\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 11:37:07.843752 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:07.843669    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbzdf\" (UniqueName: \"kubernetes.io/projected/76406e60-8b7f-41cb-ae94-c92bdccc7374-kube-api-access-sbzdf\") pod \"prometheus-k8s-0\" (UID: \"76406e60-8b7f-41cb-ae94-c92bdccc7374\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 11:37:07.844400 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:07.844340    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/76406e60-8b7f-41cb-ae94-c92bdccc7374-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"76406e60-8b7f-41cb-ae94-c92bdccc7374\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 11:37:07.845594 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:07.845560    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/76406e60-8b7f-41cb-ae94-c92bdccc7374-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"76406e60-8b7f-41cb-ae94-c92bdccc7374\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 11:37:07.956419 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:07.956377    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 11:37:08.079193 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:08.079103    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="94d3e57f-8c9b-48c6-9566-8998f808f363" path="/var/lib/kubelet/pods/94d3e57f-8c9b-48c6-9566-8998f808f363/volumes"
Jul 23 11:37:08.159075 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:08.158473    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 23 11:37:08.205703 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:08.205644    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:37:08.309215 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:08.309173    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:37:08.915114 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:37:08.915073    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76406e60_8b7f_41cb_ae94_c92bdccc7374.slice/crio-b854a45e61ccccd360e93885c6d7e84df6dfc7b1bb0fe446659b3318ef62bbdd WatchSource:0}: Error finding container b854a45e61ccccd360e93885c6d7e84df6dfc7b1bb0fe446659b3318ef62bbdd: Status 404 returned error can't find the container with id b854a45e61ccccd360e93885c6d7e84df6dfc7b1bb0fe446659b3318ef62bbdd
Jul 23 11:37:09.571342 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:09.571304    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"76406e60-8b7f-41cb-ae94-c92bdccc7374","Type":"ContainerStarted","Data":"b854a45e61ccccd360e93885c6d7e84df6dfc7b1bb0fe446659b3318ef62bbdd"}
Jul 23 11:37:10.586278 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:10.585982    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7dfbd954f7-jjdbk" event={"ID":"0b575b60-75a2-4539-bc75-08e2c621bc0b","Type":"ContainerStarted","Data":"8486fa855957225d445494952d8afd9dbf102b23219ca91ff8771c3049f611f6"}
Jul 23 11:37:10.586278 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:10.586126    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7dfbd954f7-jjdbk" event={"ID":"0b575b60-75a2-4539-bc75-08e2c621bc0b","Type":"ContainerStarted","Data":"b7a794f9ce18094518dedf0df4f98c5cdc57c6f6b765944f979deb2d2aab55ec"}
Jul 23 11:37:10.586278 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:10.586165    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7dfbd954f7-jjdbk" event={"ID":"0b575b60-75a2-4539-bc75-08e2c621bc0b","Type":"ContainerStarted","Data":"b8ed767d0f5516359ad3543db23ac107f7edbbcc3effd9e36e102d0ee01c7fac"}
Jul 23 11:37:10.589138 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:10.589064    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-8564b6f9f6-9wwdl" event={"ID":"a3b3aca3-c21e-4f24-be22-4c626e03bfbf","Type":"ContainerStarted","Data":"32c17bc68cdcdd8543f34b9e31aec87c1322e284d5b11192e47b07d1d86b04f0"}
Jul 23 11:37:10.591342 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:10.591298    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-4tn6c" event={"ID":"af96bf6b-bf0e-4dc4-8a34-71f0955c4bce","Type":"ContainerStarted","Data":"5f3b13ca151704ac2e2d871d7037cfaed72ba547bace1a9cbe7f1d4aae940551"}
Jul 23 11:37:10.592085 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:10.592053    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-4tn6c"
Jul 23 11:37:10.599732 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:10.599639    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-4tn6c"
Jul 23 11:37:10.610041 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:10.609971    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-8564b6f9f6-9wwdl" podStartSLOduration=2.120235159 podStartE2EDuration="5.609947775s" totalImagesPullingTime="3.489712616s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 11:37:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:37:10.609330125 +0000 UTC m=+85.002267917" watchObservedRunningTime="2026-07-23 11:37:10.609947775 +0000 UTC m=+85.002885564"
Jul 23 11:37:11.147317 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:11.147282    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:37:11.248405 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:11.248361    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:37:11.599033 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:11.598259    2563 generic.go:356] "Generic (PLEG): container finished" podID="76406e60-8b7f-41cb-ae94-c92bdccc7374" containerID="77f57aff0326b18fa2c85cb588953646cb75170f473749195dddc02fcf152dfe" exitCode=0
Jul 23 11:37:11.599033 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:11.598315    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"76406e60-8b7f-41cb-ae94-c92bdccc7374","Type":"ContainerDied","Data":"77f57aff0326b18fa2c85cb588953646cb75170f473749195dddc02fcf152dfe"}
Jul 23 11:37:11.629012 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:11.627890    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-4tn6c" podStartSLOduration=3.350690891 podStartE2EDuration="6.627866734s" totalImagesPullingTime="3.277175843s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 11:37:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:37:10.628443507 +0000 UTC m=+85.021381298" watchObservedRunningTime="2026-07-23 11:37:11.627866734 +0000 UTC m=+86.020804524"
Jul 23 11:37:12.608121 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:12.608016    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7dfbd954f7-jjdbk" event={"ID":"0b575b60-75a2-4539-bc75-08e2c621bc0b","Type":"ContainerStarted","Data":"49dba857250b42cfdbd03729aa3774fec78b851f3a94b16ffc0112a6cf7afb40"}
Jul 23 11:37:12.608576 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:12.608165    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7dfbd954f7-jjdbk" event={"ID":"0b575b60-75a2-4539-bc75-08e2c621bc0b","Type":"ContainerStarted","Data":"c8e5b8476f4769ba87334363738698df3ab51adf9f3c5d2d7d8877bf66dcd43a"}
Jul 23 11:37:12.608576 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:12.608212    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7dfbd954f7-jjdbk" event={"ID":"0b575b60-75a2-4539-bc75-08e2c621bc0b","Type":"ContainerStarted","Data":"475bd06767e6e105a8f184a9e8a3b523dc30a0b6bdb9f52e032e2229b79382f0"}
Jul 23 11:37:12.608576 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:12.608422    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-7dfbd954f7-jjdbk"
Jul 23 11:37:12.653614 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:12.653550    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-7dfbd954f7-jjdbk" podStartSLOduration=2.106143307 podStartE2EDuration="8.653526998s" totalImagesPullingTime="6.547383691s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 11:37:04 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:37:12.650978209 +0000 UTC m=+87.043916009" watchObservedRunningTime="2026-07-23 11:37:12.653526998 +0000 UTC m=+87.046464787"
Jul 23 11:37:15.205012 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:15.204963    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:37:15.309217 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:15.309178    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:37:15.370950 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:15.370915    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:37:16.628311 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:16.628274    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"76406e60-8b7f-41cb-ae94-c92bdccc7374","Type":"ContainerStarted","Data":"52705d199ac2d99fb722d2033724151d537ad8ae58823b5d08730eef8fea95ff"}
Jul 23 11:37:16.628861 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:16.628341    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"76406e60-8b7f-41cb-ae94-c92bdccc7374","Type":"ContainerStarted","Data":"6084befa3c3e202a4d540e080a82cbb0da37410d43c55a7ab8802204cd078dc2"}
Jul 23 11:37:16.628861 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:16.628366    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"76406e60-8b7f-41cb-ae94-c92bdccc7374","Type":"ContainerStarted","Data":"bf80d0d81b002f0c74ab456d3fc4b885e902f753fc070a2f01593d70e9712bcf"}
Jul 23 11:37:16.628861 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:16.628393    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"76406e60-8b7f-41cb-ae94-c92bdccc7374","Type":"ContainerStarted","Data":"f062774b38a89adb2cedff4c12d8b9af8e137b0d3b3f35b89bcd154f139d8648"}
Jul 23 11:37:16.628861 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:16.628416    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"76406e60-8b7f-41cb-ae94-c92bdccc7374","Type":"ContainerStarted","Data":"3ecea1875c654b22e82c6dbf2c9703db51a43d0d77c76785057a4a667da0e439"}
Jul 23 11:37:17.191586 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:17.191540    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:37:17.320732 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:17.320695    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:37:17.410293 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:17.410260    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:37:17.635309 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:17.635273    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"76406e60-8b7f-41cb-ae94-c92bdccc7374","Type":"ContainerStarted","Data":"9712f2257b63b878a221eb154a558e0450a49fc07a7223692c0a5f8514ca8989"}
Jul 23 11:37:17.667167 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:17.667096    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.9744199140000003 podStartE2EDuration="10.667073083s" totalImagesPullingTime="6.673003752s" totalInitContainerRuntime="19.649417ms" isStatefulPod=false podCreationTimestamp="2026-07-23 11:37:07 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:37:17.665576604 +0000 UTC m=+92.058514393" watchObservedRunningTime="2026-07-23 11:37:17.667073083 +0000 UTC m=+92.060010872"
Jul 23 11:37:17.957479 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:17.957385    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 11:37:18.620272 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:18.620239    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-7dfbd954f7-jjdbk"
Jul 23 11:37:23.199630 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:23.199592    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:37:23.299524 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:23.299482    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:37:26.192099 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:26.192064    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-8564b6f9f6-9wwdl"
Jul 23 11:37:26.192487 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:26.192131    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-8564b6f9f6-9wwdl"
Jul 23 11:37:35.517076 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:35.517044    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-l6jwv"
Jul 23 11:37:38.204802 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:38.204758    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:37:38.267700 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:38.267640    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:37:46.197965 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:46.197881    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-8564b6f9f6-9wwdl"
Jul 23 11:37:46.202041 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:46.202012    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-8564b6f9f6-9wwdl"
Jul 23 11:37:48.353934 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:37:48.353886    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:38:07.957620 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:07.957560    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 11:38:07.977923 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:07.977891    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 11:38:08.840336 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:08.840302    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 11:38:09.039403 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:09.039365    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:38:09.096977 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:09.096890    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:38:20.612673 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:20.612635    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:38:20.707305 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:20.707264    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:38:21.445004 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:21.444969    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:38:26.026733 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:26.026676    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 23 11:38:26.027268 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:26.026990    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="76406e60-8b7f-41cb-ae94-c92bdccc7374" containerName="prometheus" containerID="cri-o://3ecea1875c654b22e82c6dbf2c9703db51a43d0d77c76785057a4a667da0e439" gracePeriod=600
Jul 23 11:38:26.027268 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:26.027042    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="76406e60-8b7f-41cb-ae94-c92bdccc7374" containerName="config-reloader" containerID="cri-o://f062774b38a89adb2cedff4c12d8b9af8e137b0d3b3f35b89bcd154f139d8648" gracePeriod=600
Jul 23 11:38:26.027268 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:26.027045    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="76406e60-8b7f-41cb-ae94-c92bdccc7374" containerName="thanos-sidecar" containerID="cri-o://bf80d0d81b002f0c74ab456d3fc4b885e902f753fc070a2f01593d70e9712bcf" gracePeriod=600
Jul 23 11:38:26.027268 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:26.027071    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="76406e60-8b7f-41cb-ae94-c92bdccc7374" containerName="kube-rbac-proxy-web" containerID="cri-o://6084befa3c3e202a4d540e080a82cbb0da37410d43c55a7ab8802204cd078dc2" gracePeriod=600
Jul 23 11:38:26.027268 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:26.027042    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="76406e60-8b7f-41cb-ae94-c92bdccc7374" containerName="kube-rbac-proxy-thanos" containerID="cri-o://9712f2257b63b878a221eb154a558e0450a49fc07a7223692c0a5f8514ca8989" gracePeriod=600
Jul 23 11:38:26.027268 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:26.027007    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="76406e60-8b7f-41cb-ae94-c92bdccc7374" containerName="kube-rbac-proxy" containerID="cri-o://52705d199ac2d99fb722d2033724151d537ad8ae58823b5d08730eef8fea95ff" gracePeriod=600
Jul 23 11:38:26.303323 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:26.303295    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 11:38:26.385767 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:26.385734    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/76406e60-8b7f-41cb-ae94-c92bdccc7374-secret-kube-rbac-proxy\") pod \"76406e60-8b7f-41cb-ae94-c92bdccc7374\" (UID: \"76406e60-8b7f-41cb-ae94-c92bdccc7374\") "
Jul 23 11:38:26.385963 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:26.385777    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/76406e60-8b7f-41cb-ae94-c92bdccc7374-configmap-kubelet-serving-ca-bundle\") pod \"76406e60-8b7f-41cb-ae94-c92bdccc7374\" (UID: \"76406e60-8b7f-41cb-ae94-c92bdccc7374\") "
Jul 23 11:38:26.385963 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:26.385798    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/76406e60-8b7f-41cb-ae94-c92bdccc7374-web-config\") pod \"76406e60-8b7f-41cb-ae94-c92bdccc7374\" (UID: \"76406e60-8b7f-41cb-ae94-c92bdccc7374\") "
Jul 23 11:38:26.385963 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:26.385838    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/76406e60-8b7f-41cb-ae94-c92bdccc7374-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"76406e60-8b7f-41cb-ae94-c92bdccc7374\" (UID: \"76406e60-8b7f-41cb-ae94-c92bdccc7374\") "
Jul 23 11:38:26.386132 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:26.385958    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/76406e60-8b7f-41cb-ae94-c92bdccc7374-secret-grpc-tls\") pod \"76406e60-8b7f-41cb-ae94-c92bdccc7374\" (UID: \"76406e60-8b7f-41cb-ae94-c92bdccc7374\") "
Jul 23 11:38:26.386132 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:26.385999    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/76406e60-8b7f-41cb-ae94-c92bdccc7374-configmap-metrics-client-ca\") pod \"76406e60-8b7f-41cb-ae94-c92bdccc7374\" (UID: \"76406e60-8b7f-41cb-ae94-c92bdccc7374\") "
Jul 23 11:38:26.386132 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:26.386031    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/76406e60-8b7f-41cb-ae94-c92bdccc7374-prometheus-k8s-db\") pod \"76406e60-8b7f-41cb-ae94-c92bdccc7374\" (UID: \"76406e60-8b7f-41cb-ae94-c92bdccc7374\") "
Jul 23 11:38:26.386132 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:26.386057    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/76406e60-8b7f-41cb-ae94-c92bdccc7374-config\") pod \"76406e60-8b7f-41cb-ae94-c92bdccc7374\" (UID: \"76406e60-8b7f-41cb-ae94-c92bdccc7374\") "
Jul 23 11:38:26.386132 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:26.386091    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/76406e60-8b7f-41cb-ae94-c92bdccc7374-tls-assets\") pod \"76406e60-8b7f-41cb-ae94-c92bdccc7374\" (UID: \"76406e60-8b7f-41cb-ae94-c92bdccc7374\") "
Jul 23 11:38:26.386132 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:26.386117    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/76406e60-8b7f-41cb-ae94-c92bdccc7374-prometheus-k8s-rulefiles-1\") pod \"76406e60-8b7f-41cb-ae94-c92bdccc7374\" (UID: \"76406e60-8b7f-41cb-ae94-c92bdccc7374\") "
Jul 23 11:38:26.386424 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:26.386142    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/76406e60-8b7f-41cb-ae94-c92bdccc7374-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"76406e60-8b7f-41cb-ae94-c92bdccc7374\" (UID: \"76406e60-8b7f-41cb-ae94-c92bdccc7374\") "
Jul 23 11:38:26.386424 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:26.386170    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/76406e60-8b7f-41cb-ae94-c92bdccc7374-prometheus-k8s-rulefiles-0\") pod \"76406e60-8b7f-41cb-ae94-c92bdccc7374\" (UID: \"76406e60-8b7f-41cb-ae94-c92bdccc7374\") "
Jul 23 11:38:26.386424 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:26.386206    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/76406e60-8b7f-41cb-ae94-c92bdccc7374-thanos-prometheus-http-client-file\") pod \"76406e60-8b7f-41cb-ae94-c92bdccc7374\" (UID: \"76406e60-8b7f-41cb-ae94-c92bdccc7374\") "
Jul 23 11:38:26.386424 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:26.386213    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76406e60-8b7f-41cb-ae94-c92bdccc7374-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "76406e60-8b7f-41cb-ae94-c92bdccc7374" (UID: "76406e60-8b7f-41cb-ae94-c92bdccc7374"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 11:38:26.386424 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:26.386244    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/76406e60-8b7f-41cb-ae94-c92bdccc7374-secret-prometheus-k8s-tls\") pod \"76406e60-8b7f-41cb-ae94-c92bdccc7374\" (UID: \"76406e60-8b7f-41cb-ae94-c92bdccc7374\") "
Jul 23 11:38:26.386424 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:26.386365    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/76406e60-8b7f-41cb-ae94-c92bdccc7374-configmap-serving-certs-ca-bundle\") pod \"76406e60-8b7f-41cb-ae94-c92bdccc7374\" (UID: \"76406e60-8b7f-41cb-ae94-c92bdccc7374\") "
Jul 23 11:38:26.386424 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:26.386406    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/76406e60-8b7f-41cb-ae94-c92bdccc7374-secret-metrics-client-certs\") pod \"76406e60-8b7f-41cb-ae94-c92bdccc7374\" (UID: \"76406e60-8b7f-41cb-ae94-c92bdccc7374\") "
Jul 23 11:38:26.386962 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:26.386434    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/76406e60-8b7f-41cb-ae94-c92bdccc7374-config-out\") pod \"76406e60-8b7f-41cb-ae94-c92bdccc7374\" (UID: \"76406e60-8b7f-41cb-ae94-c92bdccc7374\") "
Jul 23 11:38:26.386962 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:26.386478    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/76406e60-8b7f-41cb-ae94-c92bdccc7374-prometheus-trusted-ca-bundle\") pod \"76406e60-8b7f-41cb-ae94-c92bdccc7374\" (UID: \"76406e60-8b7f-41cb-ae94-c92bdccc7374\") "
Jul 23 11:38:26.386962 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:26.386542    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76406e60-8b7f-41cb-ae94-c92bdccc7374-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "76406e60-8b7f-41cb-ae94-c92bdccc7374" (UID: "76406e60-8b7f-41cb-ae94-c92bdccc7374"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 11:38:26.386962 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:26.386560    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/76406e60-8b7f-41cb-ae94-c92bdccc7374-prometheus-k8s-rulefiles-2\") pod \"76406e60-8b7f-41cb-ae94-c92bdccc7374\" (UID: \"76406e60-8b7f-41cb-ae94-c92bdccc7374\") "
Jul 23 11:38:26.386962 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:26.386724    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sbzdf\" (UniqueName: \"kubernetes.io/projected/76406e60-8b7f-41cb-ae94-c92bdccc7374-kube-api-access-sbzdf\") pod \"76406e60-8b7f-41cb-ae94-c92bdccc7374\" (UID: \"76406e60-8b7f-41cb-ae94-c92bdccc7374\") "
Jul 23 11:38:26.388193 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:26.388134    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76406e60-8b7f-41cb-ae94-c92bdccc7374-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "76406e60-8b7f-41cb-ae94-c92bdccc7374" (UID: "76406e60-8b7f-41cb-ae94-c92bdccc7374"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 11:38:26.388604 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:26.388572    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76406e60-8b7f-41cb-ae94-c92bdccc7374-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "76406e60-8b7f-41cb-ae94-c92bdccc7374" (UID: "76406e60-8b7f-41cb-ae94-c92bdccc7374"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 11:38:26.390225 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:26.390188    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/76406e60-8b7f-41cb-ae94-c92bdccc7374-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "76406e60-8b7f-41cb-ae94-c92bdccc7374" (UID: "76406e60-8b7f-41cb-ae94-c92bdccc7374"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 11:38:26.390225 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:26.390206    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/76406e60-8b7f-41cb-ae94-c92bdccc7374-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "76406e60-8b7f-41cb-ae94-c92bdccc7374" (UID: "76406e60-8b7f-41cb-ae94-c92bdccc7374"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 11:38:26.390440 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:26.390408    2563 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/76406e60-8b7f-41cb-ae94-c92bdccc7374-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 23 11:38:26.390559 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:26.390450    2563 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/76406e60-8b7f-41cb-ae94-c92bdccc7374-secret-grpc-tls\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 23 11:38:26.390559 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:26.390468    2563 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/76406e60-8b7f-41cb-ae94-c92bdccc7374-configmap-metrics-client-ca\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 23 11:38:26.390559 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:26.390489    2563 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/76406e60-8b7f-41cb-ae94-c92bdccc7374-prometheus-k8s-db\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 23 11:38:26.390559 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:26.390505    2563 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/76406e60-8b7f-41cb-ae94-c92bdccc7374-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 23 11:38:26.390559 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:26.390510    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76406e60-8b7f-41cb-ae94-c92bdccc7374-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "76406e60-8b7f-41cb-ae94-c92bdccc7374" (UID: "76406e60-8b7f-41cb-ae94-c92bdccc7374"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 11:38:26.390559 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:26.390520    2563 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/76406e60-8b7f-41cb-ae94-c92bdccc7374-secret-prometheus-k8s-tls\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 23 11:38:26.390897 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:26.390595    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76406e60-8b7f-41cb-ae94-c92bdccc7374-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "76406e60-8b7f-41cb-ae94-c92bdccc7374" (UID: "76406e60-8b7f-41cb-ae94-c92bdccc7374"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 11:38:26.390956 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:26.390921    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76406e60-8b7f-41cb-ae94-c92bdccc7374-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "76406e60-8b7f-41cb-ae94-c92bdccc7374" (UID: "76406e60-8b7f-41cb-ae94-c92bdccc7374"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 11:38:26.391371 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:26.391312    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/76406e60-8b7f-41cb-ae94-c92bdccc7374-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "76406e60-8b7f-41cb-ae94-c92bdccc7374" (UID: "76406e60-8b7f-41cb-ae94-c92bdccc7374"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 11:38:26.392327 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:26.391824    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/76406e60-8b7f-41cb-ae94-c92bdccc7374-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "76406e60-8b7f-41cb-ae94-c92bdccc7374" (UID: "76406e60-8b7f-41cb-ae94-c92bdccc7374"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 11:38:26.392327 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:26.391876    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76406e60-8b7f-41cb-ae94-c92bdccc7374-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "76406e60-8b7f-41cb-ae94-c92bdccc7374" (UID: "76406e60-8b7f-41cb-ae94-c92bdccc7374"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 11:38:26.392327 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:26.392026    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/76406e60-8b7f-41cb-ae94-c92bdccc7374-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "76406e60-8b7f-41cb-ae94-c92bdccc7374" (UID: "76406e60-8b7f-41cb-ae94-c92bdccc7374"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 11:38:26.392539 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:26.392371    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/76406e60-8b7f-41cb-ae94-c92bdccc7374-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "76406e60-8b7f-41cb-ae94-c92bdccc7374" (UID: "76406e60-8b7f-41cb-ae94-c92bdccc7374"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 11:38:26.396772 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:26.396701    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/76406e60-8b7f-41cb-ae94-c92bdccc7374-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "76406e60-8b7f-41cb-ae94-c92bdccc7374" (UID: "76406e60-8b7f-41cb-ae94-c92bdccc7374"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 11:38:26.396772 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:26.396723    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76406e60-8b7f-41cb-ae94-c92bdccc7374-kube-api-access-sbzdf" (OuterVolumeSpecName: "kube-api-access-sbzdf") pod "76406e60-8b7f-41cb-ae94-c92bdccc7374" (UID: "76406e60-8b7f-41cb-ae94-c92bdccc7374"). InnerVolumeSpecName "kube-api-access-sbzdf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 11:38:26.396952 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:26.396742    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/76406e60-8b7f-41cb-ae94-c92bdccc7374-config" (OuterVolumeSpecName: "config") pod "76406e60-8b7f-41cb-ae94-c92bdccc7374" (UID: "76406e60-8b7f-41cb-ae94-c92bdccc7374"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 11:38:26.396952 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:26.396776    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76406e60-8b7f-41cb-ae94-c92bdccc7374-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "76406e60-8b7f-41cb-ae94-c92bdccc7374" (UID: "76406e60-8b7f-41cb-ae94-c92bdccc7374"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 11:38:26.398292 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:26.398259    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76406e60-8b7f-41cb-ae94-c92bdccc7374-config-out" (OuterVolumeSpecName: "config-out") pod "76406e60-8b7f-41cb-ae94-c92bdccc7374" (UID: "76406e60-8b7f-41cb-ae94-c92bdccc7374"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 11:38:26.402144 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:26.402120    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/76406e60-8b7f-41cb-ae94-c92bdccc7374-web-config" (OuterVolumeSpecName: "web-config") pod "76406e60-8b7f-41cb-ae94-c92bdccc7374" (UID: "76406e60-8b7f-41cb-ae94-c92bdccc7374"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 11:38:26.491014 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:26.490969    2563 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/76406e60-8b7f-41cb-ae94-c92bdccc7374-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 23 11:38:26.491014 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:26.491009    2563 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/76406e60-8b7f-41cb-ae94-c92bdccc7374-secret-metrics-client-certs\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 23 11:38:26.491244 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:26.491024    2563 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/76406e60-8b7f-41cb-ae94-c92bdccc7374-config-out\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 23 11:38:26.491244 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:26.491039    2563 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/76406e60-8b7f-41cb-ae94-c92bdccc7374-prometheus-trusted-ca-bundle\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 23 11:38:26.491244 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:26.491052    2563 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/76406e60-8b7f-41cb-ae94-c92bdccc7374-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 23 11:38:26.491244 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:26.491066    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sbzdf\" (UniqueName: \"kubernetes.io/projected/76406e60-8b7f-41cb-ae94-c92bdccc7374-kube-api-access-sbzdf\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 23 11:38:26.491244 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:26.491077    2563 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/76406e60-8b7f-41cb-ae94-c92bdccc7374-secret-kube-rbac-proxy\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 23 11:38:26.491244 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:26.491089    2563 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/76406e60-8b7f-41cb-ae94-c92bdccc7374-web-config\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 23 11:38:26.491244 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:26.491101    2563 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/76406e60-8b7f-41cb-ae94-c92bdccc7374-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 23 11:38:26.491244 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:26.491114    2563 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/76406e60-8b7f-41cb-ae94-c92bdccc7374-config\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 23 11:38:26.491244 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:26.491125    2563 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/76406e60-8b7f-41cb-ae94-c92bdccc7374-tls-assets\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 23 11:38:26.491244 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:26.491137    2563 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/76406e60-8b7f-41cb-ae94-c92bdccc7374-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 23 11:38:26.491244 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:26.491149    2563 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/76406e60-8b7f-41cb-ae94-c92bdccc7374-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 23 11:38:26.491244 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:26.491162    2563 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/76406e60-8b7f-41cb-ae94-c92bdccc7374-thanos-prometheus-http-client-file\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 23 11:38:26.894785 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:26.894751    2563 generic.go:356] "Generic (PLEG): container finished" podID="76406e60-8b7f-41cb-ae94-c92bdccc7374" containerID="9712f2257b63b878a221eb154a558e0450a49fc07a7223692c0a5f8514ca8989" exitCode=0
Jul 23 11:38:26.894785 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:26.894777    2563 generic.go:356] "Generic (PLEG): container finished" podID="76406e60-8b7f-41cb-ae94-c92bdccc7374" containerID="52705d199ac2d99fb722d2033724151d537ad8ae58823b5d08730eef8fea95ff" exitCode=0
Jul 23 11:38:26.894785 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:26.894784    2563 generic.go:356] "Generic (PLEG): container finished" podID="76406e60-8b7f-41cb-ae94-c92bdccc7374" containerID="6084befa3c3e202a4d540e080a82cbb0da37410d43c55a7ab8802204cd078dc2" exitCode=0
Jul 23 11:38:26.894785 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:26.894790    2563 generic.go:356] "Generic (PLEG): container finished" podID="76406e60-8b7f-41cb-ae94-c92bdccc7374" containerID="bf80d0d81b002f0c74ab456d3fc4b885e902f753fc070a2f01593d70e9712bcf" exitCode=0
Jul 23 11:38:26.894785 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:26.894795    2563 generic.go:356] "Generic (PLEG): container finished" podID="76406e60-8b7f-41cb-ae94-c92bdccc7374" containerID="f062774b38a89adb2cedff4c12d8b9af8e137b0d3b3f35b89bcd154f139d8648" exitCode=0
Jul 23 11:38:26.895088 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:26.894801    2563 generic.go:356] "Generic (PLEG): container finished" podID="76406e60-8b7f-41cb-ae94-c92bdccc7374" containerID="3ecea1875c654b22e82c6dbf2c9703db51a43d0d77c76785057a4a667da0e439" exitCode=0
Jul 23 11:38:26.895088 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:26.894831    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"76406e60-8b7f-41cb-ae94-c92bdccc7374","Type":"ContainerDied","Data":"9712f2257b63b878a221eb154a558e0450a49fc07a7223692c0a5f8514ca8989"}
Jul 23 11:38:26.895088 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:26.894845    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 11:38:26.895088 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:26.894929    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"76406e60-8b7f-41cb-ae94-c92bdccc7374","Type":"ContainerDied","Data":"52705d199ac2d99fb722d2033724151d537ad8ae58823b5d08730eef8fea95ff"}
Jul 23 11:38:26.895088 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:26.894958    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"76406e60-8b7f-41cb-ae94-c92bdccc7374","Type":"ContainerDied","Data":"6084befa3c3e202a4d540e080a82cbb0da37410d43c55a7ab8802204cd078dc2"}
Jul 23 11:38:26.895088 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:26.894983    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"76406e60-8b7f-41cb-ae94-c92bdccc7374","Type":"ContainerDied","Data":"bf80d0d81b002f0c74ab456d3fc4b885e902f753fc070a2f01593d70e9712bcf"}
Jul 23 11:38:26.895088 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:26.894990    2563 scope.go:117] "RemoveContainer" containerID="9712f2257b63b878a221eb154a558e0450a49fc07a7223692c0a5f8514ca8989"
Jul 23 11:38:26.895088 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:26.895007    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"76406e60-8b7f-41cb-ae94-c92bdccc7374","Type":"ContainerDied","Data":"f062774b38a89adb2cedff4c12d8b9af8e137b0d3b3f35b89bcd154f139d8648"}
Jul 23 11:38:26.895088 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:26.895032    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"76406e60-8b7f-41cb-ae94-c92bdccc7374","Type":"ContainerDied","Data":"3ecea1875c654b22e82c6dbf2c9703db51a43d0d77c76785057a4a667da0e439"}
Jul 23 11:38:26.895088 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:26.895057    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"76406e60-8b7f-41cb-ae94-c92bdccc7374","Type":"ContainerDied","Data":"b854a45e61ccccd360e93885c6d7e84df6dfc7b1bb0fe446659b3318ef62bbdd"}
Jul 23 11:38:26.903255 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:26.903062    2563 scope.go:117] "RemoveContainer" containerID="52705d199ac2d99fb722d2033724151d537ad8ae58823b5d08730eef8fea95ff"
Jul 23 11:38:26.910613 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:26.910594    2563 scope.go:117] "RemoveContainer" containerID="6084befa3c3e202a4d540e080a82cbb0da37410d43c55a7ab8802204cd078dc2"
Jul 23 11:38:26.918828 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:26.918796    2563 scope.go:117] "RemoveContainer" containerID="bf80d0d81b002f0c74ab456d3fc4b885e902f753fc070a2f01593d70e9712bcf"
Jul 23 11:38:26.920130 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:26.920107    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 23 11:38:26.927197 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:26.927166    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 23 11:38:26.927769 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:26.927747    2563 scope.go:117] "RemoveContainer" containerID="f062774b38a89adb2cedff4c12d8b9af8e137b0d3b3f35b89bcd154f139d8648"
Jul 23 11:38:26.935195 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:26.935171    2563 scope.go:117] "RemoveContainer" containerID="3ecea1875c654b22e82c6dbf2c9703db51a43d0d77c76785057a4a667da0e439"
Jul 23 11:38:26.942946 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:26.942922    2563 scope.go:117] "RemoveContainer" containerID="77f57aff0326b18fa2c85cb588953646cb75170f473749195dddc02fcf152dfe"
Jul 23 11:38:26.950407 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:26.950388    2563 scope.go:117] "RemoveContainer" containerID="9712f2257b63b878a221eb154a558e0450a49fc07a7223692c0a5f8514ca8989"
Jul 23 11:38:26.950751 ip-10-0-141-229 kubenswrapper[2563]: E0723 11:38:26.950727    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9712f2257b63b878a221eb154a558e0450a49fc07a7223692c0a5f8514ca8989\": container with ID starting with 9712f2257b63b878a221eb154a558e0450a49fc07a7223692c0a5f8514ca8989 not found: ID does not exist" containerID="9712f2257b63b878a221eb154a558e0450a49fc07a7223692c0a5f8514ca8989"
Jul 23 11:38:26.950850 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:26.950759    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9712f2257b63b878a221eb154a558e0450a49fc07a7223692c0a5f8514ca8989"} err="failed to get container status \"9712f2257b63b878a221eb154a558e0450a49fc07a7223692c0a5f8514ca8989\": rpc error: code = NotFound desc = could not find container \"9712f2257b63b878a221eb154a558e0450a49fc07a7223692c0a5f8514ca8989\": container with ID starting with 9712f2257b63b878a221eb154a558e0450a49fc07a7223692c0a5f8514ca8989 not found: ID does not exist"
Jul 23 11:38:26.950850 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:26.950780    2563 scope.go:117] "RemoveContainer" containerID="52705d199ac2d99fb722d2033724151d537ad8ae58823b5d08730eef8fea95ff"
Jul 23 11:38:26.951081 ip-10-0-141-229 kubenswrapper[2563]: E0723 11:38:26.951062    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"52705d199ac2d99fb722d2033724151d537ad8ae58823b5d08730eef8fea95ff\": container with ID starting with 52705d199ac2d99fb722d2033724151d537ad8ae58823b5d08730eef8fea95ff not found: ID does not exist" containerID="52705d199ac2d99fb722d2033724151d537ad8ae58823b5d08730eef8fea95ff"
Jul 23 11:38:26.951155 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:26.951090    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52705d199ac2d99fb722d2033724151d537ad8ae58823b5d08730eef8fea95ff"} err="failed to get container status \"52705d199ac2d99fb722d2033724151d537ad8ae58823b5d08730eef8fea95ff\": rpc error: code = NotFound desc = could not find container \"52705d199ac2d99fb722d2033724151d537ad8ae58823b5d08730eef8fea95ff\": container with ID starting with 52705d199ac2d99fb722d2033724151d537ad8ae58823b5d08730eef8fea95ff not found: ID does not exist"
Jul 23 11:38:26.951155 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:26.951112    2563 scope.go:117] "RemoveContainer" containerID="6084befa3c3e202a4d540e080a82cbb0da37410d43c55a7ab8802204cd078dc2"
Jul 23 11:38:26.951362 ip-10-0-141-229 kubenswrapper[2563]: E0723 11:38:26.951347    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6084befa3c3e202a4d540e080a82cbb0da37410d43c55a7ab8802204cd078dc2\": container with ID starting with 6084befa3c3e202a4d540e080a82cbb0da37410d43c55a7ab8802204cd078dc2 not found: ID does not exist" containerID="6084befa3c3e202a4d540e080a82cbb0da37410d43c55a7ab8802204cd078dc2"
Jul 23 11:38:26.951402 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:26.951366    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6084befa3c3e202a4d540e080a82cbb0da37410d43c55a7ab8802204cd078dc2"} err="failed to get container status \"6084befa3c3e202a4d540e080a82cbb0da37410d43c55a7ab8802204cd078dc2\": rpc error: code = NotFound desc = could not find container \"6084befa3c3e202a4d540e080a82cbb0da37410d43c55a7ab8802204cd078dc2\": container with ID starting with 6084befa3c3e202a4d540e080a82cbb0da37410d43c55a7ab8802204cd078dc2 not found: ID does not exist"
Jul 23 11:38:26.951402 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:26.951379    2563 scope.go:117] "RemoveContainer" containerID="bf80d0d81b002f0c74ab456d3fc4b885e902f753fc070a2f01593d70e9712bcf"
Jul 23 11:38:26.951620 ip-10-0-141-229 kubenswrapper[2563]: E0723 11:38:26.951603    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf80d0d81b002f0c74ab456d3fc4b885e902f753fc070a2f01593d70e9712bcf\": container with ID starting with bf80d0d81b002f0c74ab456d3fc4b885e902f753fc070a2f01593d70e9712bcf not found: ID does not exist" containerID="bf80d0d81b002f0c74ab456d3fc4b885e902f753fc070a2f01593d70e9712bcf"
Jul 23 11:38:26.951709 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:26.951627    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf80d0d81b002f0c74ab456d3fc4b885e902f753fc070a2f01593d70e9712bcf"} err="failed to get container status \"bf80d0d81b002f0c74ab456d3fc4b885e902f753fc070a2f01593d70e9712bcf\": rpc error: code = NotFound desc = could not find container \"bf80d0d81b002f0c74ab456d3fc4b885e902f753fc070a2f01593d70e9712bcf\": container with ID starting with bf80d0d81b002f0c74ab456d3fc4b885e902f753fc070a2f01593d70e9712bcf not found: ID does not exist"
Jul 23 11:38:26.951709 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:26.951645    2563 scope.go:117] "RemoveContainer" containerID="f062774b38a89adb2cedff4c12d8b9af8e137b0d3b3f35b89bcd154f139d8648"
Jul 23 11:38:26.951929 ip-10-0-141-229 kubenswrapper[2563]: E0723 11:38:26.951909    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f062774b38a89adb2cedff4c12d8b9af8e137b0d3b3f35b89bcd154f139d8648\": container with ID starting with f062774b38a89adb2cedff4c12d8b9af8e137b0d3b3f35b89bcd154f139d8648 not found: ID does not exist" containerID="f062774b38a89adb2cedff4c12d8b9af8e137b0d3b3f35b89bcd154f139d8648"
Jul 23 11:38:26.951968 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:26.951935    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f062774b38a89adb2cedff4c12d8b9af8e137b0d3b3f35b89bcd154f139d8648"} err="failed to get container status \"f062774b38a89adb2cedff4c12d8b9af8e137b0d3b3f35b89bcd154f139d8648\": rpc error: code = NotFound desc = could not find container \"f062774b38a89adb2cedff4c12d8b9af8e137b0d3b3f35b89bcd154f139d8648\": container with ID starting with f062774b38a89adb2cedff4c12d8b9af8e137b0d3b3f35b89bcd154f139d8648 not found: ID does not exist"
Jul 23 11:38:26.951968 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:26.951952    2563 scope.go:117] "RemoveContainer" containerID="3ecea1875c654b22e82c6dbf2c9703db51a43d0d77c76785057a4a667da0e439"
Jul 23 11:38:26.952207 ip-10-0-141-229 kubenswrapper[2563]: E0723 11:38:26.952187    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ecea1875c654b22e82c6dbf2c9703db51a43d0d77c76785057a4a667da0e439\": container with ID starting with 3ecea1875c654b22e82c6dbf2c9703db51a43d0d77c76785057a4a667da0e439 not found: ID does not exist" containerID="3ecea1875c654b22e82c6dbf2c9703db51a43d0d77c76785057a4a667da0e439"
Jul 23 11:38:26.952273 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:26.952213    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ecea1875c654b22e82c6dbf2c9703db51a43d0d77c76785057a4a667da0e439"} err="failed to get container status \"3ecea1875c654b22e82c6dbf2c9703db51a43d0d77c76785057a4a667da0e439\": rpc error: code = NotFound desc = could not find container \"3ecea1875c654b22e82c6dbf2c9703db51a43d0d77c76785057a4a667da0e439\": container with ID starting with 3ecea1875c654b22e82c6dbf2c9703db51a43d0d77c76785057a4a667da0e439 not found: ID does not exist"
Jul 23 11:38:26.952273 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:26.952232    2563 scope.go:117] "RemoveContainer" containerID="77f57aff0326b18fa2c85cb588953646cb75170f473749195dddc02fcf152dfe"
Jul 23 11:38:26.952470 ip-10-0-141-229 kubenswrapper[2563]: E0723 11:38:26.952456    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77f57aff0326b18fa2c85cb588953646cb75170f473749195dddc02fcf152dfe\": container with ID starting with 77f57aff0326b18fa2c85cb588953646cb75170f473749195dddc02fcf152dfe not found: ID does not exist" containerID="77f57aff0326b18fa2c85cb588953646cb75170f473749195dddc02fcf152dfe"
Jul 23 11:38:26.952530 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:26.952476    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77f57aff0326b18fa2c85cb588953646cb75170f473749195dddc02fcf152dfe"} err="failed to get container status \"77f57aff0326b18fa2c85cb588953646cb75170f473749195dddc02fcf152dfe\": rpc error: code = NotFound desc = could not find container \"77f57aff0326b18fa2c85cb588953646cb75170f473749195dddc02fcf152dfe\": container with ID starting with 77f57aff0326b18fa2c85cb588953646cb75170f473749195dddc02fcf152dfe not found: ID does not exist"
Jul 23 11:38:26.952530 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:26.952493    2563 scope.go:117] "RemoveContainer" containerID="9712f2257b63b878a221eb154a558e0450a49fc07a7223692c0a5f8514ca8989"
Jul 23 11:38:26.952778 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:26.952754    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9712f2257b63b878a221eb154a558e0450a49fc07a7223692c0a5f8514ca8989"} err="failed to get container status \"9712f2257b63b878a221eb154a558e0450a49fc07a7223692c0a5f8514ca8989\": rpc error: code = NotFound desc = could not find container \"9712f2257b63b878a221eb154a558e0450a49fc07a7223692c0a5f8514ca8989\": container with ID starting with 9712f2257b63b878a221eb154a558e0450a49fc07a7223692c0a5f8514ca8989 not found: ID does not exist"
Jul 23 11:38:26.952860 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:26.952781    2563 scope.go:117] "RemoveContainer" containerID="52705d199ac2d99fb722d2033724151d537ad8ae58823b5d08730eef8fea95ff"
Jul 23 11:38:26.953088 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:26.953057    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52705d199ac2d99fb722d2033724151d537ad8ae58823b5d08730eef8fea95ff"} err="failed to get container status \"52705d199ac2d99fb722d2033724151d537ad8ae58823b5d08730eef8fea95ff\": rpc error: code = NotFound desc = could not find container \"52705d199ac2d99fb722d2033724151d537ad8ae58823b5d08730eef8fea95ff\": container with ID starting with 52705d199ac2d99fb722d2033724151d537ad8ae58823b5d08730eef8fea95ff not found: ID does not exist"
Jul 23 11:38:26.953245 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:26.953097    2563 scope.go:117] "RemoveContainer" containerID="6084befa3c3e202a4d540e080a82cbb0da37410d43c55a7ab8802204cd078dc2"
Jul 23 11:38:26.953415 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:26.953377    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6084befa3c3e202a4d540e080a82cbb0da37410d43c55a7ab8802204cd078dc2"} err="failed to get container status \"6084befa3c3e202a4d540e080a82cbb0da37410d43c55a7ab8802204cd078dc2\": rpc error: code = NotFound desc = could not find container \"6084befa3c3e202a4d540e080a82cbb0da37410d43c55a7ab8802204cd078dc2\": container with ID starting with 6084befa3c3e202a4d540e080a82cbb0da37410d43c55a7ab8802204cd078dc2 not found: ID does not exist"
Jul 23 11:38:26.953415 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:26.953409    2563 scope.go:117] "RemoveContainer" containerID="bf80d0d81b002f0c74ab456d3fc4b885e902f753fc070a2f01593d70e9712bcf"
Jul 23 11:38:26.953755 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:26.953710    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf80d0d81b002f0c74ab456d3fc4b885e902f753fc070a2f01593d70e9712bcf"} err="failed to get container status \"bf80d0d81b002f0c74ab456d3fc4b885e902f753fc070a2f01593d70e9712bcf\": rpc error: code = NotFound desc = could not find container \"bf80d0d81b002f0c74ab456d3fc4b885e902f753fc070a2f01593d70e9712bcf\": container with ID starting with bf80d0d81b002f0c74ab456d3fc4b885e902f753fc070a2f01593d70e9712bcf not found: ID does not exist"
Jul 23 11:38:26.953755 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:26.953737    2563 scope.go:117] "RemoveContainer" containerID="f062774b38a89adb2cedff4c12d8b9af8e137b0d3b3f35b89bcd154f139d8648"
Jul 23 11:38:26.954047 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:26.954024    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f062774b38a89adb2cedff4c12d8b9af8e137b0d3b3f35b89bcd154f139d8648"} err="failed to get container status \"f062774b38a89adb2cedff4c12d8b9af8e137b0d3b3f35b89bcd154f139d8648\": rpc error: code = NotFound desc = could not find container \"f062774b38a89adb2cedff4c12d8b9af8e137b0d3b3f35b89bcd154f139d8648\": container with ID starting with f062774b38a89adb2cedff4c12d8b9af8e137b0d3b3f35b89bcd154f139d8648 not found: ID does not exist"
Jul 23 11:38:26.954136 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:26.954048    2563 scope.go:117] "RemoveContainer" containerID="3ecea1875c654b22e82c6dbf2c9703db51a43d0d77c76785057a4a667da0e439"
Jul 23 11:38:26.954315 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:26.954297    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ecea1875c654b22e82c6dbf2c9703db51a43d0d77c76785057a4a667da0e439"} err="failed to get container status \"3ecea1875c654b22e82c6dbf2c9703db51a43d0d77c76785057a4a667da0e439\": rpc error: code = NotFound desc = could not find container \"3ecea1875c654b22e82c6dbf2c9703db51a43d0d77c76785057a4a667da0e439\": container with ID starting with 3ecea1875c654b22e82c6dbf2c9703db51a43d0d77c76785057a4a667da0e439 not found: ID does not exist"
Jul 23 11:38:26.954374 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:26.954315    2563 scope.go:117] "RemoveContainer" containerID="77f57aff0326b18fa2c85cb588953646cb75170f473749195dddc02fcf152dfe"
Jul 23 11:38:26.954534 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:26.954514    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77f57aff0326b18fa2c85cb588953646cb75170f473749195dddc02fcf152dfe"} err="failed to get container status \"77f57aff0326b18fa2c85cb588953646cb75170f473749195dddc02fcf152dfe\": rpc error: code = NotFound desc = could not find container \"77f57aff0326b18fa2c85cb588953646cb75170f473749195dddc02fcf152dfe\": container with ID starting with 77f57aff0326b18fa2c85cb588953646cb75170f473749195dddc02fcf152dfe not found: ID does not exist"
Jul 23 11:38:26.954588 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:26.954535    2563 scope.go:117] "RemoveContainer" containerID="9712f2257b63b878a221eb154a558e0450a49fc07a7223692c0a5f8514ca8989"
Jul 23 11:38:26.954809 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:26.954789    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9712f2257b63b878a221eb154a558e0450a49fc07a7223692c0a5f8514ca8989"} err="failed to get container status \"9712f2257b63b878a221eb154a558e0450a49fc07a7223692c0a5f8514ca8989\": rpc error: code = NotFound desc = could not find container \"9712f2257b63b878a221eb154a558e0450a49fc07a7223692c0a5f8514ca8989\": container with ID starting with 9712f2257b63b878a221eb154a558e0450a49fc07a7223692c0a5f8514ca8989 not found: ID does not exist"
Jul 23 11:38:26.954875 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:26.954810    2563 scope.go:117] "RemoveContainer" containerID="52705d199ac2d99fb722d2033724151d537ad8ae58823b5d08730eef8fea95ff"
Jul 23 11:38:26.955031 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:26.955007    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 23 11:38:26.955077 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:26.955050    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52705d199ac2d99fb722d2033724151d537ad8ae58823b5d08730eef8fea95ff"} err="failed to get container status \"52705d199ac2d99fb722d2033724151d537ad8ae58823b5d08730eef8fea95ff\": rpc error: code = NotFound desc = could not find container \"52705d199ac2d99fb722d2033724151d537ad8ae58823b5d08730eef8fea95ff\": container with ID starting with 52705d199ac2d99fb722d2033724151d537ad8ae58823b5d08730eef8fea95ff not found: ID does not exist"
Jul 23 11:38:26.955077 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:26.955067    2563 scope.go:117] "RemoveContainer" containerID="6084befa3c3e202a4d540e080a82cbb0da37410d43c55a7ab8802204cd078dc2"
Jul 23 11:38:26.955294 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:26.955278    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6084befa3c3e202a4d540e080a82cbb0da37410d43c55a7ab8802204cd078dc2"} err="failed to get container status \"6084befa3c3e202a4d540e080a82cbb0da37410d43c55a7ab8802204cd078dc2\": rpc error: code = NotFound desc = could not find container \"6084befa3c3e202a4d540e080a82cbb0da37410d43c55a7ab8802204cd078dc2\": container with ID starting with 6084befa3c3e202a4d540e080a82cbb0da37410d43c55a7ab8802204cd078dc2 not found: ID does not exist"
Jul 23 11:38:26.955342 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:26.955293    2563 scope.go:117] "RemoveContainer" containerID="bf80d0d81b002f0c74ab456d3fc4b885e902f753fc070a2f01593d70e9712bcf"
Jul 23 11:38:26.955342 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:26.955308    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="76406e60-8b7f-41cb-ae94-c92bdccc7374" containerName="thanos-sidecar"
Jul 23 11:38:26.955342 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:26.955322    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="76406e60-8b7f-41cb-ae94-c92bdccc7374" containerName="thanos-sidecar"
Jul 23 11:38:26.955342 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:26.955334    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="76406e60-8b7f-41cb-ae94-c92bdccc7374" containerName="config-reloader"
Jul 23 11:38:26.955342 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:26.955340    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="76406e60-8b7f-41cb-ae94-c92bdccc7374" containerName="config-reloader"
Jul 23 11:38:26.955496 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:26.955349    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="76406e60-8b7f-41cb-ae94-c92bdccc7374" containerName="prometheus"
Jul 23 11:38:26.955496 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:26.955355    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="76406e60-8b7f-41cb-ae94-c92bdccc7374" containerName="prometheus"
Jul 23 11:38:26.955496 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:26.955360    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="76406e60-8b7f-41cb-ae94-c92bdccc7374" containerName="kube-rbac-proxy-web"
Jul 23 11:38:26.955496 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:26.955365    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="76406e60-8b7f-41cb-ae94-c92bdccc7374" containerName="kube-rbac-proxy-web"
Jul 23 11:38:26.955496 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:26.955384    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="76406e60-8b7f-41cb-ae94-c92bdccc7374" containerName="init-config-reloader"
Jul 23 11:38:26.955496 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:26.955393    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="76406e60-8b7f-41cb-ae94-c92bdccc7374" containerName="init-config-reloader"
Jul 23 11:38:26.955496 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:26.955399    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="76406e60-8b7f-41cb-ae94-c92bdccc7374" containerName="kube-rbac-proxy-thanos"
Jul 23 11:38:26.955496 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:26.955405    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="76406e60-8b7f-41cb-ae94-c92bdccc7374" containerName="kube-rbac-proxy-thanos"
Jul 23 11:38:26.955496 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:26.955414    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="76406e60-8b7f-41cb-ae94-c92bdccc7374" containerName="kube-rbac-proxy"
Jul 23 11:38:26.955496 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:26.955419    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="76406e60-8b7f-41cb-ae94-c92bdccc7374" containerName="kube-rbac-proxy"
Jul 23 11:38:26.955496 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:26.955478    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="76406e60-8b7f-41cb-ae94-c92bdccc7374" containerName="prometheus"
Jul 23 11:38:26.955496 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:26.955486    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="76406e60-8b7f-41cb-ae94-c92bdccc7374" containerName="config-reloader"
Jul 23 11:38:26.955496 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:26.955495    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="76406e60-8b7f-41cb-ae94-c92bdccc7374" containerName="kube-rbac-proxy-web"
Jul 23 11:38:26.955913 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:26.955505    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="76406e60-8b7f-41cb-ae94-c92bdccc7374" containerName="thanos-sidecar"
Jul 23 11:38:26.955913 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:26.955511    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="76406e60-8b7f-41cb-ae94-c92bdccc7374" containerName="kube-rbac-proxy"
Jul 23 11:38:26.955913 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:26.955516    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="76406e60-8b7f-41cb-ae94-c92bdccc7374" containerName="kube-rbac-proxy-thanos"
Jul 23 11:38:26.955913 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:26.955525    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf80d0d81b002f0c74ab456d3fc4b885e902f753fc070a2f01593d70e9712bcf"} err="failed to get container status \"bf80d0d81b002f0c74ab456d3fc4b885e902f753fc070a2f01593d70e9712bcf\": rpc error: code = NotFound desc = could not find container \"bf80d0d81b002f0c74ab456d3fc4b885e902f753fc070a2f01593d70e9712bcf\": container with ID starting with bf80d0d81b002f0c74ab456d3fc4b885e902f753fc070a2f01593d70e9712bcf not found: ID does not exist"
Jul 23 11:38:26.955913 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:26.955543    2563 scope.go:117] "RemoveContainer" containerID="f062774b38a89adb2cedff4c12d8b9af8e137b0d3b3f35b89bcd154f139d8648"
Jul 23 11:38:26.955913 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:26.955814    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f062774b38a89adb2cedff4c12d8b9af8e137b0d3b3f35b89bcd154f139d8648"} err="failed to get container status \"f062774b38a89adb2cedff4c12d8b9af8e137b0d3b3f35b89bcd154f139d8648\": rpc error: code = NotFound desc = could not find container \"f062774b38a89adb2cedff4c12d8b9af8e137b0d3b3f35b89bcd154f139d8648\": container with ID starting with f062774b38a89adb2cedff4c12d8b9af8e137b0d3b3f35b89bcd154f139d8648 not found: ID does not exist"
Jul 23 11:38:26.955913 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:26.955829    2563 scope.go:117] "RemoveContainer" containerID="3ecea1875c654b22e82c6dbf2c9703db51a43d0d77c76785057a4a667da0e439"
Jul 23 11:38:26.956152 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:26.956076    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ecea1875c654b22e82c6dbf2c9703db51a43d0d77c76785057a4a667da0e439"} err="failed to get container status \"3ecea1875c654b22e82c6dbf2c9703db51a43d0d77c76785057a4a667da0e439\": rpc error: code = NotFound desc = could not find container \"3ecea1875c654b22e82c6dbf2c9703db51a43d0d77c76785057a4a667da0e439\": container with ID starting with 3ecea1875c654b22e82c6dbf2c9703db51a43d0d77c76785057a4a667da0e439 not found: ID does not exist"
Jul 23 11:38:26.956152 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:26.956097    2563 scope.go:117] "RemoveContainer" containerID="77f57aff0326b18fa2c85cb588953646cb75170f473749195dddc02fcf152dfe"
Jul 23 11:38:26.956350 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:26.956328    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77f57aff0326b18fa2c85cb588953646cb75170f473749195dddc02fcf152dfe"} err="failed to get container status \"77f57aff0326b18fa2c85cb588953646cb75170f473749195dddc02fcf152dfe\": rpc error: code = NotFound desc = could not find container \"77f57aff0326b18fa2c85cb588953646cb75170f473749195dddc02fcf152dfe\": container with ID starting with 77f57aff0326b18fa2c85cb588953646cb75170f473749195dddc02fcf152dfe not found: ID does not exist"
Jul 23 11:38:26.956417 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:26.956352    2563 scope.go:117] "RemoveContainer" containerID="9712f2257b63b878a221eb154a558e0450a49fc07a7223692c0a5f8514ca8989"
Jul 23 11:38:26.956604 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:26.956583    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9712f2257b63b878a221eb154a558e0450a49fc07a7223692c0a5f8514ca8989"} err="failed to get container status \"9712f2257b63b878a221eb154a558e0450a49fc07a7223692c0a5f8514ca8989\": rpc error: code = NotFound desc = could not find container \"9712f2257b63b878a221eb154a558e0450a49fc07a7223692c0a5f8514ca8989\": container with ID starting with 9712f2257b63b878a221eb154a558e0450a49fc07a7223692c0a5f8514ca8989 not found: ID does not exist"
Jul 23 11:38:26.956647 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:26.956606    2563 scope.go:117] "RemoveContainer" containerID="52705d199ac2d99fb722d2033724151d537ad8ae58823b5d08730eef8fea95ff"
Jul 23 11:38:26.956833 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:26.956814    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52705d199ac2d99fb722d2033724151d537ad8ae58823b5d08730eef8fea95ff"} err="failed to get container status \"52705d199ac2d99fb722d2033724151d537ad8ae58823b5d08730eef8fea95ff\": rpc error: code = NotFound desc = could not find container \"52705d199ac2d99fb722d2033724151d537ad8ae58823b5d08730eef8fea95ff\": container with ID starting with 52705d199ac2d99fb722d2033724151d537ad8ae58823b5d08730eef8fea95ff not found: ID does not exist"
Jul 23 11:38:26.956907 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:26.956835    2563 scope.go:117] "RemoveContainer" containerID="6084befa3c3e202a4d540e080a82cbb0da37410d43c55a7ab8802204cd078dc2"
Jul 23 11:38:26.957084 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:26.957059    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6084befa3c3e202a4d540e080a82cbb0da37410d43c55a7ab8802204cd078dc2"} err="failed to get container status \"6084befa3c3e202a4d540e080a82cbb0da37410d43c55a7ab8802204cd078dc2\": rpc error: code = NotFound desc = could not find container \"6084befa3c3e202a4d540e080a82cbb0da37410d43c55a7ab8802204cd078dc2\": container with ID starting with 6084befa3c3e202a4d540e080a82cbb0da37410d43c55a7ab8802204cd078dc2 not found: ID does not exist"
Jul 23 11:38:26.957131 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:26.957084    2563 scope.go:117] "RemoveContainer" containerID="bf80d0d81b002f0c74ab456d3fc4b885e902f753fc070a2f01593d70e9712bcf"
Jul 23 11:38:26.957317 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:26.957299    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf80d0d81b002f0c74ab456d3fc4b885e902f753fc070a2f01593d70e9712bcf"} err="failed to get container status \"bf80d0d81b002f0c74ab456d3fc4b885e902f753fc070a2f01593d70e9712bcf\": rpc error: code = NotFound desc = could not find container \"bf80d0d81b002f0c74ab456d3fc4b885e902f753fc070a2f01593d70e9712bcf\": container with ID starting with bf80d0d81b002f0c74ab456d3fc4b885e902f753fc070a2f01593d70e9712bcf not found: ID does not exist"
Jul 23 11:38:26.957361 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:26.957318    2563 scope.go:117] "RemoveContainer" containerID="f062774b38a89adb2cedff4c12d8b9af8e137b0d3b3f35b89bcd154f139d8648"
Jul 23 11:38:26.957534 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:26.957510    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f062774b38a89adb2cedff4c12d8b9af8e137b0d3b3f35b89bcd154f139d8648"} err="failed to get container status \"f062774b38a89adb2cedff4c12d8b9af8e137b0d3b3f35b89bcd154f139d8648\": rpc error: code = NotFound desc = could not find container \"f062774b38a89adb2cedff4c12d8b9af8e137b0d3b3f35b89bcd154f139d8648\": container with ID starting with f062774b38a89adb2cedff4c12d8b9af8e137b0d3b3f35b89bcd154f139d8648 not found: ID does not exist"
Jul 23 11:38:26.957604 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:26.957537    2563 scope.go:117] "RemoveContainer" containerID="3ecea1875c654b22e82c6dbf2c9703db51a43d0d77c76785057a4a667da0e439"
Jul 23 11:38:26.957779 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:26.957763    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ecea1875c654b22e82c6dbf2c9703db51a43d0d77c76785057a4a667da0e439"} err="failed to get container status \"3ecea1875c654b22e82c6dbf2c9703db51a43d0d77c76785057a4a667da0e439\": rpc error: code = NotFound desc = could not find container \"3ecea1875c654b22e82c6dbf2c9703db51a43d0d77c76785057a4a667da0e439\": container with ID starting with 3ecea1875c654b22e82c6dbf2c9703db51a43d0d77c76785057a4a667da0e439 not found: ID does not exist"
Jul 23 11:38:26.957852 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:26.957781    2563 scope.go:117] "RemoveContainer" containerID="77f57aff0326b18fa2c85cb588953646cb75170f473749195dddc02fcf152dfe"
Jul 23 11:38:26.957979 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:26.957959    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77f57aff0326b18fa2c85cb588953646cb75170f473749195dddc02fcf152dfe"} err="failed to get container status \"77f57aff0326b18fa2c85cb588953646cb75170f473749195dddc02fcf152dfe\": rpc error: code = NotFound desc = could not find container \"77f57aff0326b18fa2c85cb588953646cb75170f473749195dddc02fcf152dfe\": container with ID starting with 77f57aff0326b18fa2c85cb588953646cb75170f473749195dddc02fcf152dfe not found: ID does not exist"
Jul 23 11:38:26.958042 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:26.957981    2563 scope.go:117] "RemoveContainer" containerID="9712f2257b63b878a221eb154a558e0450a49fc07a7223692c0a5f8514ca8989"
Jul 23 11:38:26.958195 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:26.958176    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9712f2257b63b878a221eb154a558e0450a49fc07a7223692c0a5f8514ca8989"} err="failed to get container status \"9712f2257b63b878a221eb154a558e0450a49fc07a7223692c0a5f8514ca8989\": rpc error: code = NotFound desc = could not find container \"9712f2257b63b878a221eb154a558e0450a49fc07a7223692c0a5f8514ca8989\": container with ID starting with 9712f2257b63b878a221eb154a558e0450a49fc07a7223692c0a5f8514ca8989 not found: ID does not exist"
Jul 23 11:38:26.958238 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:26.958196    2563 scope.go:117] "RemoveContainer" containerID="52705d199ac2d99fb722d2033724151d537ad8ae58823b5d08730eef8fea95ff"
Jul 23 11:38:26.958388 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:26.958370    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52705d199ac2d99fb722d2033724151d537ad8ae58823b5d08730eef8fea95ff"} err="failed to get container status \"52705d199ac2d99fb722d2033724151d537ad8ae58823b5d08730eef8fea95ff\": rpc error: code = NotFound desc = could not find container \"52705d199ac2d99fb722d2033724151d537ad8ae58823b5d08730eef8fea95ff\": container with ID starting with 52705d199ac2d99fb722d2033724151d537ad8ae58823b5d08730eef8fea95ff not found: ID does not exist"
Jul 23 11:38:26.958450 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:26.958389    2563 scope.go:117] "RemoveContainer" containerID="6084befa3c3e202a4d540e080a82cbb0da37410d43c55a7ab8802204cd078dc2"
Jul 23 11:38:26.958624 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:26.958608    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6084befa3c3e202a4d540e080a82cbb0da37410d43c55a7ab8802204cd078dc2"} err="failed to get container status \"6084befa3c3e202a4d540e080a82cbb0da37410d43c55a7ab8802204cd078dc2\": rpc error: code = NotFound desc = could not find container \"6084befa3c3e202a4d540e080a82cbb0da37410d43c55a7ab8802204cd078dc2\": container with ID starting with 6084befa3c3e202a4d540e080a82cbb0da37410d43c55a7ab8802204cd078dc2 not found: ID does not exist"
Jul 23 11:38:26.958669 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:26.958624    2563 scope.go:117] "RemoveContainer" containerID="bf80d0d81b002f0c74ab456d3fc4b885e902f753fc070a2f01593d70e9712bcf"
Jul 23 11:38:26.958837 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:26.958820    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf80d0d81b002f0c74ab456d3fc4b885e902f753fc070a2f01593d70e9712bcf"} err="failed to get container status \"bf80d0d81b002f0c74ab456d3fc4b885e902f753fc070a2f01593d70e9712bcf\": rpc error: code = NotFound desc = could not find container \"bf80d0d81b002f0c74ab456d3fc4b885e902f753fc070a2f01593d70e9712bcf\": container with ID starting with bf80d0d81b002f0c74ab456d3fc4b885e902f753fc070a2f01593d70e9712bcf not found: ID does not exist"
Jul 23 11:38:26.958889 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:26.958837    2563 scope.go:117] "RemoveContainer" containerID="f062774b38a89adb2cedff4c12d8b9af8e137b0d3b3f35b89bcd154f139d8648"
Jul 23 11:38:26.959067 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:26.959048    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f062774b38a89adb2cedff4c12d8b9af8e137b0d3b3f35b89bcd154f139d8648"} err="failed to get container status \"f062774b38a89adb2cedff4c12d8b9af8e137b0d3b3f35b89bcd154f139d8648\": rpc error: code = NotFound desc = could not find container \"f062774b38a89adb2cedff4c12d8b9af8e137b0d3b3f35b89bcd154f139d8648\": container with ID starting with f062774b38a89adb2cedff4c12d8b9af8e137b0d3b3f35b89bcd154f139d8648 not found: ID does not exist"
Jul 23 11:38:26.959116 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:26.959067    2563 scope.go:117] "RemoveContainer" containerID="3ecea1875c654b22e82c6dbf2c9703db51a43d0d77c76785057a4a667da0e439"
Jul 23 11:38:26.959309 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:26.959288    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ecea1875c654b22e82c6dbf2c9703db51a43d0d77c76785057a4a667da0e439"} err="failed to get container status \"3ecea1875c654b22e82c6dbf2c9703db51a43d0d77c76785057a4a667da0e439\": rpc error: code = NotFound desc = could not find container \"3ecea1875c654b22e82c6dbf2c9703db51a43d0d77c76785057a4a667da0e439\": container with ID starting with 3ecea1875c654b22e82c6dbf2c9703db51a43d0d77c76785057a4a667da0e439 not found: ID does not exist"
Jul 23 11:38:26.959353 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:26.959311    2563 scope.go:117] "RemoveContainer" containerID="77f57aff0326b18fa2c85cb588953646cb75170f473749195dddc02fcf152dfe"
Jul 23 11:38:26.959532 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:26.959515    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77f57aff0326b18fa2c85cb588953646cb75170f473749195dddc02fcf152dfe"} err="failed to get container status \"77f57aff0326b18fa2c85cb588953646cb75170f473749195dddc02fcf152dfe\": rpc error: code = NotFound desc = could not find container \"77f57aff0326b18fa2c85cb588953646cb75170f473749195dddc02fcf152dfe\": container with ID starting with 77f57aff0326b18fa2c85cb588953646cb75170f473749195dddc02fcf152dfe not found: ID does not exist"
Jul 23 11:38:26.959577 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:26.959532    2563 scope.go:117] "RemoveContainer" containerID="9712f2257b63b878a221eb154a558e0450a49fc07a7223692c0a5f8514ca8989"
Jul 23 11:38:26.959766 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:26.959743    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9712f2257b63b878a221eb154a558e0450a49fc07a7223692c0a5f8514ca8989"} err="failed to get container status \"9712f2257b63b878a221eb154a558e0450a49fc07a7223692c0a5f8514ca8989\": rpc error: code = NotFound desc = could not find container \"9712f2257b63b878a221eb154a558e0450a49fc07a7223692c0a5f8514ca8989\": container with ID starting with 9712f2257b63b878a221eb154a558e0450a49fc07a7223692c0a5f8514ca8989 not found: ID does not exist"
Jul 23 11:38:26.959820 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:26.959766    2563 scope.go:117] "RemoveContainer" containerID="52705d199ac2d99fb722d2033724151d537ad8ae58823b5d08730eef8fea95ff"
Jul 23 11:38:26.959990 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:26.959963    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52705d199ac2d99fb722d2033724151d537ad8ae58823b5d08730eef8fea95ff"} err="failed to get container status \"52705d199ac2d99fb722d2033724151d537ad8ae58823b5d08730eef8fea95ff\": rpc error: code = NotFound desc = could not find container \"52705d199ac2d99fb722d2033724151d537ad8ae58823b5d08730eef8fea95ff\": container with ID starting with 52705d199ac2d99fb722d2033724151d537ad8ae58823b5d08730eef8fea95ff not found: ID does not exist"
Jul 23 11:38:26.959990 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:26.959989    2563 scope.go:117] "RemoveContainer" containerID="6084befa3c3e202a4d540e080a82cbb0da37410d43c55a7ab8802204cd078dc2"
Jul 23 11:38:26.960110 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:26.959970    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 11:38:26.960185 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:26.960164    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6084befa3c3e202a4d540e080a82cbb0da37410d43c55a7ab8802204cd078dc2"} err="failed to get container status \"6084befa3c3e202a4d540e080a82cbb0da37410d43c55a7ab8802204cd078dc2\": rpc error: code = NotFound desc = could not find container \"6084befa3c3e202a4d540e080a82cbb0da37410d43c55a7ab8802204cd078dc2\": container with ID starting with 6084befa3c3e202a4d540e080a82cbb0da37410d43c55a7ab8802204cd078dc2 not found: ID does not exist"
Jul 23 11:38:26.960234 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:26.960186    2563 scope.go:117] "RemoveContainer" containerID="bf80d0d81b002f0c74ab456d3fc4b885e902f753fc070a2f01593d70e9712bcf"
Jul 23 11:38:26.960735 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:26.960712    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf80d0d81b002f0c74ab456d3fc4b885e902f753fc070a2f01593d70e9712bcf"} err="failed to get container status \"bf80d0d81b002f0c74ab456d3fc4b885e902f753fc070a2f01593d70e9712bcf\": rpc error: code = NotFound desc = could not find container \"bf80d0d81b002f0c74ab456d3fc4b885e902f753fc070a2f01593d70e9712bcf\": container with ID starting with bf80d0d81b002f0c74ab456d3fc4b885e902f753fc070a2f01593d70e9712bcf not found: ID does not exist"
Jul 23 11:38:26.960802 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:26.960735    2563 scope.go:117] "RemoveContainer" containerID="f062774b38a89adb2cedff4c12d8b9af8e137b0d3b3f35b89bcd154f139d8648"
Jul 23 11:38:26.960993 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:26.960975    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f062774b38a89adb2cedff4c12d8b9af8e137b0d3b3f35b89bcd154f139d8648"} err="failed to get container status \"f062774b38a89adb2cedff4c12d8b9af8e137b0d3b3f35b89bcd154f139d8648\": rpc error: code = NotFound desc = could not find container \"f062774b38a89adb2cedff4c12d8b9af8e137b0d3b3f35b89bcd154f139d8648\": container with ID starting with f062774b38a89adb2cedff4c12d8b9af8e137b0d3b3f35b89bcd154f139d8648 not found: ID does not exist"
Jul 23 11:38:26.961051 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:26.960995    2563 scope.go:117] "RemoveContainer" containerID="3ecea1875c654b22e82c6dbf2c9703db51a43d0d77c76785057a4a667da0e439"
Jul 23 11:38:26.961246 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:26.961221    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ecea1875c654b22e82c6dbf2c9703db51a43d0d77c76785057a4a667da0e439"} err="failed to get container status \"3ecea1875c654b22e82c6dbf2c9703db51a43d0d77c76785057a4a667da0e439\": rpc error: code = NotFound desc = could not find container \"3ecea1875c654b22e82c6dbf2c9703db51a43d0d77c76785057a4a667da0e439\": container with ID starting with 3ecea1875c654b22e82c6dbf2c9703db51a43d0d77c76785057a4a667da0e439 not found: ID does not exist"
Jul 23 11:38:26.961246 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:26.961246    2563 scope.go:117] "RemoveContainer" containerID="77f57aff0326b18fa2c85cb588953646cb75170f473749195dddc02fcf152dfe"
Jul 23 11:38:26.961488 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:26.961467    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77f57aff0326b18fa2c85cb588953646cb75170f473749195dddc02fcf152dfe"} err="failed to get container status \"77f57aff0326b18fa2c85cb588953646cb75170f473749195dddc02fcf152dfe\": rpc error: code = NotFound desc = could not find container \"77f57aff0326b18fa2c85cb588953646cb75170f473749195dddc02fcf152dfe\": container with ID starting with 77f57aff0326b18fa2c85cb588953646cb75170f473749195dddc02fcf152dfe not found: ID does not exist"
Jul 23 11:38:26.962620 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:26.962599    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 23 11:38:26.962769 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:26.962629    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 23 11:38:26.962769 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:26.962635    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 23 11:38:26.962769 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:26.962671    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-pljtt\""
Jul 23 11:38:26.962769 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:26.962749    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 23 11:38:26.962977 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:26.962960    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 23 11:38:26.963060 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:26.963045    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 23 11:38:26.963104 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:26.963070    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 23 11:38:26.963104 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:26.963075    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 23 11:38:26.963182 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:26.963148    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 23 11:38:26.963228 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:26.963201    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 23 11:38:26.963720 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:26.963705    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 23 11:38:26.963786 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:26.963763    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-f27n8jjki1o2b\""
Jul 23 11:38:26.964021 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:26.964003    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 23 11:38:26.966985 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:26.966943    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 23 11:38:26.970236 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:26.970215    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 23 11:38:26.975313 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:26.975290    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 23 11:38:26.994221 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:26.994172    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1de24f23-477c-4fbb-964a-7802ee86612e-web-config\") pod \"prometheus-k8s-0\" (UID: \"1de24f23-477c-4fbb-964a-7802ee86612e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 11:38:26.994221 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:26.994221    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/1de24f23-477c-4fbb-964a-7802ee86612e-config\") pod \"prometheus-k8s-0\" (UID: \"1de24f23-477c-4fbb-964a-7802ee86612e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 11:38:26.994470 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:26.994246    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/1de24f23-477c-4fbb-964a-7802ee86612e-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"1de24f23-477c-4fbb-964a-7802ee86612e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 11:38:26.994470 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:26.994271    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1de24f23-477c-4fbb-964a-7802ee86612e-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1de24f23-477c-4fbb-964a-7802ee86612e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 11:38:26.994470 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:26.994300    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1de24f23-477c-4fbb-964a-7802ee86612e-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"1de24f23-477c-4fbb-964a-7802ee86612e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 11:38:26.994470 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:26.994324    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1de24f23-477c-4fbb-964a-7802ee86612e-config-out\") pod \"prometheus-k8s-0\" (UID: \"1de24f23-477c-4fbb-964a-7802ee86612e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 11:38:26.994470 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:26.994344    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/1de24f23-477c-4fbb-964a-7802ee86612e-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"1de24f23-477c-4fbb-964a-7802ee86612e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 11:38:26.994470 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:26.994412    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1de24f23-477c-4fbb-964a-7802ee86612e-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"1de24f23-477c-4fbb-964a-7802ee86612e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 11:38:26.994470 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:26.994447    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1de24f23-477c-4fbb-964a-7802ee86612e-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1de24f23-477c-4fbb-964a-7802ee86612e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 11:38:26.994906 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:26.994476    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/1de24f23-477c-4fbb-964a-7802ee86612e-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"1de24f23-477c-4fbb-964a-7802ee86612e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 11:38:26.994906 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:26.994491    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/1de24f23-477c-4fbb-964a-7802ee86612e-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"1de24f23-477c-4fbb-964a-7802ee86612e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 11:38:26.994906 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:26.994540    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/1de24f23-477c-4fbb-964a-7802ee86612e-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"1de24f23-477c-4fbb-964a-7802ee86612e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 11:38:26.994906 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:26.994586    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/1de24f23-477c-4fbb-964a-7802ee86612e-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"1de24f23-477c-4fbb-964a-7802ee86612e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 11:38:26.994906 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:26.994620    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1de24f23-477c-4fbb-964a-7802ee86612e-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"1de24f23-477c-4fbb-964a-7802ee86612e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 11:38:26.994906 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:26.994746    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/1de24f23-477c-4fbb-964a-7802ee86612e-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"1de24f23-477c-4fbb-964a-7802ee86612e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 11:38:26.994906 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:26.994791    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1de24f23-477c-4fbb-964a-7802ee86612e-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1de24f23-477c-4fbb-964a-7802ee86612e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 11:38:26.994906 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:26.994827    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1de24f23-477c-4fbb-964a-7802ee86612e-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"1de24f23-477c-4fbb-964a-7802ee86612e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 11:38:26.994906 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:26.994854    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/1de24f23-477c-4fbb-964a-7802ee86612e-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"1de24f23-477c-4fbb-964a-7802ee86612e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 11:38:26.994906 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:26.994889    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwnlk\" (UniqueName: \"kubernetes.io/projected/1de24f23-477c-4fbb-964a-7802ee86612e-kube-api-access-pwnlk\") pod \"prometheus-k8s-0\" (UID: \"1de24f23-477c-4fbb-964a-7802ee86612e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 11:38:26.995196 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:26.994925    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/1de24f23-477c-4fbb-964a-7802ee86612e-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"1de24f23-477c-4fbb-964a-7802ee86612e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 11:38:27.096145 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:27.096103    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1de24f23-477c-4fbb-964a-7802ee86612e-web-config\") pod \"prometheus-k8s-0\" (UID: \"1de24f23-477c-4fbb-964a-7802ee86612e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 11:38:27.096145 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:27.096151    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/1de24f23-477c-4fbb-964a-7802ee86612e-config\") pod \"prometheus-k8s-0\" (UID: \"1de24f23-477c-4fbb-964a-7802ee86612e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 11:38:27.096715 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:27.096168    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/1de24f23-477c-4fbb-964a-7802ee86612e-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"1de24f23-477c-4fbb-964a-7802ee86612e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 11:38:27.096715 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:27.096185    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1de24f23-477c-4fbb-964a-7802ee86612e-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1de24f23-477c-4fbb-964a-7802ee86612e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 11:38:27.096715 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:27.096207    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1de24f23-477c-4fbb-964a-7802ee86612e-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"1de24f23-477c-4fbb-964a-7802ee86612e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 11:38:27.096715 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:27.096225    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1de24f23-477c-4fbb-964a-7802ee86612e-config-out\") pod \"prometheus-k8s-0\" (UID: \"1de24f23-477c-4fbb-964a-7802ee86612e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 11:38:27.096715 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:27.096244    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/1de24f23-477c-4fbb-964a-7802ee86612e-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"1de24f23-477c-4fbb-964a-7802ee86612e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 11:38:27.096976 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:27.096834    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1de24f23-477c-4fbb-964a-7802ee86612e-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"1de24f23-477c-4fbb-964a-7802ee86612e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 11:38:27.096976 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:27.096895    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1de24f23-477c-4fbb-964a-7802ee86612e-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1de24f23-477c-4fbb-964a-7802ee86612e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 11:38:27.096976 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:27.096924    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/1de24f23-477c-4fbb-964a-7802ee86612e-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"1de24f23-477c-4fbb-964a-7802ee86612e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 11:38:27.096976 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:27.096950    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/1de24f23-477c-4fbb-964a-7802ee86612e-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"1de24f23-477c-4fbb-964a-7802ee86612e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 11:38:27.097336 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:27.096981    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/1de24f23-477c-4fbb-964a-7802ee86612e-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"1de24f23-477c-4fbb-964a-7802ee86612e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 11:38:27.097336 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:27.097012    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/1de24f23-477c-4fbb-964a-7802ee86612e-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"1de24f23-477c-4fbb-964a-7802ee86612e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 11:38:27.097336 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:27.097054    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1de24f23-477c-4fbb-964a-7802ee86612e-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"1de24f23-477c-4fbb-964a-7802ee86612e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 11:38:27.097336 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:27.097264    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/1de24f23-477c-4fbb-964a-7802ee86612e-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"1de24f23-477c-4fbb-964a-7802ee86612e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 11:38:27.097336 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:27.097304    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1de24f23-477c-4fbb-964a-7802ee86612e-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1de24f23-477c-4fbb-964a-7802ee86612e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 11:38:27.097336 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:27.097313    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1de24f23-477c-4fbb-964a-7802ee86612e-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1de24f23-477c-4fbb-964a-7802ee86612e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 11:38:27.097615 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:27.097355    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1de24f23-477c-4fbb-964a-7802ee86612e-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"1de24f23-477c-4fbb-964a-7802ee86612e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 11:38:27.097615 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:27.097387    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/1de24f23-477c-4fbb-964a-7802ee86612e-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"1de24f23-477c-4fbb-964a-7802ee86612e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 11:38:27.097615 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:27.097410    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwnlk\" (UniqueName: \"kubernetes.io/projected/1de24f23-477c-4fbb-964a-7802ee86612e-kube-api-access-pwnlk\") pod \"prometheus-k8s-0\" (UID: \"1de24f23-477c-4fbb-964a-7802ee86612e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 11:38:27.097615 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:27.097448    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/1de24f23-477c-4fbb-964a-7802ee86612e-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"1de24f23-477c-4fbb-964a-7802ee86612e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 11:38:27.097615 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:27.097597    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/1de24f23-477c-4fbb-964a-7802ee86612e-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"1de24f23-477c-4fbb-964a-7802ee86612e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 11:38:27.098235 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:27.098208    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1de24f23-477c-4fbb-964a-7802ee86612e-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"1de24f23-477c-4fbb-964a-7802ee86612e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 11:38:27.099867 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:27.098818    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1de24f23-477c-4fbb-964a-7802ee86612e-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1de24f23-477c-4fbb-964a-7802ee86612e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 11:38:27.099867 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:27.099419    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1de24f23-477c-4fbb-964a-7802ee86612e-config-out\") pod \"prometheus-k8s-0\" (UID: \"1de24f23-477c-4fbb-964a-7802ee86612e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 11:38:27.099867 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:27.099556    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1de24f23-477c-4fbb-964a-7802ee86612e-web-config\") pod \"prometheus-k8s-0\" (UID: \"1de24f23-477c-4fbb-964a-7802ee86612e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 11:38:27.099867 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:27.099590    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/1de24f23-477c-4fbb-964a-7802ee86612e-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"1de24f23-477c-4fbb-964a-7802ee86612e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 11:38:27.099867 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:27.099745    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/1de24f23-477c-4fbb-964a-7802ee86612e-config\") pod \"prometheus-k8s-0\" (UID: \"1de24f23-477c-4fbb-964a-7802ee86612e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 11:38:27.099867 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:27.099777    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1de24f23-477c-4fbb-964a-7802ee86612e-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"1de24f23-477c-4fbb-964a-7802ee86612e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 11:38:27.100243 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:27.099954    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/1de24f23-477c-4fbb-964a-7802ee86612e-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"1de24f23-477c-4fbb-964a-7802ee86612e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 11:38:27.100243 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:27.100153    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/1de24f23-477c-4fbb-964a-7802ee86612e-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"1de24f23-477c-4fbb-964a-7802ee86612e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 11:38:27.101260 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:27.100406    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1de24f23-477c-4fbb-964a-7802ee86612e-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1de24f23-477c-4fbb-964a-7802ee86612e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 11:38:27.101260 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:27.100733    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/1de24f23-477c-4fbb-964a-7802ee86612e-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"1de24f23-477c-4fbb-964a-7802ee86612e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 11:38:27.101260 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:27.101157    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/1de24f23-477c-4fbb-964a-7802ee86612e-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"1de24f23-477c-4fbb-964a-7802ee86612e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 11:38:27.102556 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:27.102528    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/1de24f23-477c-4fbb-964a-7802ee86612e-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"1de24f23-477c-4fbb-964a-7802ee86612e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 11:38:27.102657 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:27.102611    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1de24f23-477c-4fbb-964a-7802ee86612e-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"1de24f23-477c-4fbb-964a-7802ee86612e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 11:38:27.102767 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:27.102672    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/1de24f23-477c-4fbb-964a-7802ee86612e-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"1de24f23-477c-4fbb-964a-7802ee86612e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 11:38:27.102832 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:27.102790    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1de24f23-477c-4fbb-964a-7802ee86612e-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"1de24f23-477c-4fbb-964a-7802ee86612e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 11:38:27.103069 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:27.103048    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/1de24f23-477c-4fbb-964a-7802ee86612e-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"1de24f23-477c-4fbb-964a-7802ee86612e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 11:38:27.108507 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:27.108481    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwnlk\" (UniqueName: \"kubernetes.io/projected/1de24f23-477c-4fbb-964a-7802ee86612e-kube-api-access-pwnlk\") pod \"prometheus-k8s-0\" (UID: \"1de24f23-477c-4fbb-964a-7802ee86612e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 11:38:27.271021 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:27.270910    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 11:38:27.419366 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:27.419335    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 23 11:38:27.422533 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:38:27.422505    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1de24f23_477c_4fbb_964a_7802ee86612e.slice/crio-e437a745ec31b91d61a02d3f114f43706930c2763c7fe3dc55056245791d2312 WatchSource:0}: Error finding container e437a745ec31b91d61a02d3f114f43706930c2763c7fe3dc55056245791d2312: Status 404 returned error can't find the container with id e437a745ec31b91d61a02d3f114f43706930c2763c7fe3dc55056245791d2312
Jul 23 11:38:27.900077 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:27.900038    2563 generic.go:356] "Generic (PLEG): container finished" podID="1de24f23-477c-4fbb-964a-7802ee86612e" containerID="136b6f5e4b902973bc2984467b11fd8f38811c944ae7fdce71b8b68b975ec453" exitCode=0
Jul 23 11:38:27.900230 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:27.900128    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1de24f23-477c-4fbb-964a-7802ee86612e","Type":"ContainerDied","Data":"136b6f5e4b902973bc2984467b11fd8f38811c944ae7fdce71b8b68b975ec453"}
Jul 23 11:38:27.900230 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:27.900204    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1de24f23-477c-4fbb-964a-7802ee86612e","Type":"ContainerStarted","Data":"e437a745ec31b91d61a02d3f114f43706930c2763c7fe3dc55056245791d2312"}
Jul 23 11:38:28.078890 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:28.078848    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="76406e60-8b7f-41cb-ae94-c92bdccc7374" path="/var/lib/kubelet/pods/76406e60-8b7f-41cb-ae94-c92bdccc7374/volumes"
Jul 23 11:38:28.908532 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:28.908492    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1de24f23-477c-4fbb-964a-7802ee86612e","Type":"ContainerStarted","Data":"0f3c470b9640972dec2ee015257ec28f16d3c41ffe052449f3635728a9bd9c87"}
Jul 23 11:38:28.909011 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:28.908577    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1de24f23-477c-4fbb-964a-7802ee86612e","Type":"ContainerStarted","Data":"97487f07b0a57387174a8f54c561f5ae73a62fc41af181b76fcf88a317631158"}
Jul 23 11:38:28.909011 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:28.908624    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1de24f23-477c-4fbb-964a-7802ee86612e","Type":"ContainerStarted","Data":"223aed1999892a9c8d328527b21a4315d0d3781106f3a89c4f0f75cef8f79c27"}
Jul 23 11:38:28.909011 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:28.908663    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1de24f23-477c-4fbb-964a-7802ee86612e","Type":"ContainerStarted","Data":"6b199afc6c021199c81c92949eab3479b7796f0d6f2aa3e4b547c8feae514530"}
Jul 23 11:38:28.909011 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:28.908729    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1de24f23-477c-4fbb-964a-7802ee86612e","Type":"ContainerStarted","Data":"293bb6131b16bdd00c3da1caf51b66ec0718991217330ee787a0967c48ffbadb"}
Jul 23 11:38:28.909011 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:28.908765    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1de24f23-477c-4fbb-964a-7802ee86612e","Type":"ContainerStarted","Data":"0edca8843d1e52bf642eeb64b6b4f312b062435e4f29afe8a5f421cd8aa75f20"}
Jul 23 11:38:28.937550 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:28.937482    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.912962275 podStartE2EDuration="2.937456471s" totalImagesPullingTime="0s" totalInitContainerRuntime="24.494196ms" isStatefulPod=false podCreationTimestamp="2026-07-23 11:38:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:38:28.935257569 +0000 UTC m=+163.328195356" watchObservedRunningTime="2026-07-23 11:38:28.937456471 +0000 UTC m=+163.330394258"
Jul 23 11:38:30.199781 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:30.199726    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:38:30.274587 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:30.274543    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:38:32.271252 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:32.271190    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 11:38:41.498252 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:41.498156    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:38:41.559245 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:38:41.559201    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:39:27.271069 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:39:27.271022    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 11:39:27.287909 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:39:27.287878    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 11:39:28.148498 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:39:28.148464    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 23 11:39:35.883271 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:39:35.883231    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:39:35.945844 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:39:35.945810    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:39:38.278843 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:39:38.278805    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:39:38.347260 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:39:38.347208    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:39:40.906392 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:39:40.906349    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:39:47.933633 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:39:47.933584    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:39:48.033320 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:39:48.033274    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:39:48.770943 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:39:48.770906    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:39:48.868634 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:39:48.868594    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:39:48.927935 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:39:48.927895    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:39:53.234224 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:39:53.234194    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:39:53.332436 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:39:53.332401    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:39:58.637622 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:39:58.637590    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:39:58.731960 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:39:58.731922    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:40:15.061663 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:40:15.061580    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:40:15.140337 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:40:15.140305    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:40:25.067762 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:40:25.067716    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:40:25.181321 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:40:25.181281    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:40:45.956448 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:40:45.956413    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nhm9p_93649ac9-9062-46c0-adaa-1048d471211a/ovn-acl-logging/0.log"
Jul 23 11:40:45.961035 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:40:45.961006    2563 kubelet.go:1648] "Image garbage collection succeeded"
Jul 23 11:40:45.963185 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:40:45.963162    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nhm9p_93649ac9-9062-46c0-adaa-1048d471211a/ovn-acl-logging/0.log"
Jul 23 11:40:46.835428 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:40:46.835391    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:40:48.169880 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:40:48.169843    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:40:48.838977 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:40:48.838941    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-h4wb2"]
Jul 23 11:40:48.841025 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:40:48.841008    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-h4wb2"
Jul 23 11:40:48.844679 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:40:48.844654    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\""
Jul 23 11:40:48.844825 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:40:48.844725    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\""
Jul 23 11:40:48.844893 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:40:48.844872    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-mgxjx\""
Jul 23 11:40:48.866166 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:40:48.866136    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-h4wb2"]
Jul 23 11:40:48.992165 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:40:48.992122    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4641d770-fcfe-448f-8c05-deb00274e96c-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-h4wb2\" (UID: \"4641d770-fcfe-448f-8c05-deb00274e96c\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-h4wb2"
Jul 23 11:40:48.992350 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:40:48.992233    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qcwz\" (UniqueName: \"kubernetes.io/projected/4641d770-fcfe-448f-8c05-deb00274e96c-kube-api-access-9qcwz\") pod \"cert-manager-operator-controller-manager-55c444bf86-h4wb2\" (UID: \"4641d770-fcfe-448f-8c05-deb00274e96c\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-h4wb2"
Jul 23 11:40:49.093322 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:40:49.093234    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qcwz\" (UniqueName: \"kubernetes.io/projected/4641d770-fcfe-448f-8c05-deb00274e96c-kube-api-access-9qcwz\") pod \"cert-manager-operator-controller-manager-55c444bf86-h4wb2\" (UID: \"4641d770-fcfe-448f-8c05-deb00274e96c\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-h4wb2"
Jul 23 11:40:49.093322 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:40:49.093295    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4641d770-fcfe-448f-8c05-deb00274e96c-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-h4wb2\" (UID: \"4641d770-fcfe-448f-8c05-deb00274e96c\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-h4wb2"
Jul 23 11:40:49.093656 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:40:49.093640    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4641d770-fcfe-448f-8c05-deb00274e96c-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-h4wb2\" (UID: \"4641d770-fcfe-448f-8c05-deb00274e96c\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-h4wb2"
Jul 23 11:40:49.102573 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:40:49.102536    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qcwz\" (UniqueName: \"kubernetes.io/projected/4641d770-fcfe-448f-8c05-deb00274e96c-kube-api-access-9qcwz\") pod \"cert-manager-operator-controller-manager-55c444bf86-h4wb2\" (UID: \"4641d770-fcfe-448f-8c05-deb00274e96c\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-h4wb2"
Jul 23 11:40:49.150279 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:40:49.150236    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-h4wb2"
Jul 23 11:40:49.288543 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:40:49.288506    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-h4wb2"]
Jul 23 11:40:49.293370 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:40:49.293343    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4641d770_fcfe_448f_8c05_deb00274e96c.slice/crio-89a6c63a3b2288207efbc6cdff7ae3598fd455a6bbe939100fabe5b8379b4791 WatchSource:0}: Error finding container 89a6c63a3b2288207efbc6cdff7ae3598fd455a6bbe939100fabe5b8379b4791: Status 404 returned error can't find the container with id 89a6c63a3b2288207efbc6cdff7ae3598fd455a6bbe939100fabe5b8379b4791
Jul 23 11:40:49.295856 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:40:49.295837    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 23 11:40:49.435630 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:40:49.435593    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-h4wb2" event={"ID":"4641d770-fcfe-448f-8c05-deb00274e96c","Type":"ContainerStarted","Data":"89a6c63a3b2288207efbc6cdff7ae3598fd455a6bbe939100fabe5b8379b4791"}
Jul 23 11:40:50.436334 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:40:50.436228    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:40:50.500630 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:40:50.500574    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:40:52.451243 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:40:52.451205    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-h4wb2" event={"ID":"4641d770-fcfe-448f-8c05-deb00274e96c","Type":"ContainerStarted","Data":"6dac9873da88952809a2caa6520dfe2408c6443ef3ab77938b6baef8b81c0ece"}
Jul 23 11:40:52.482926 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:40:52.482861    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-h4wb2" podStartSLOduration=1.794782449 podStartE2EDuration="4.482841338s" totalImagesPullingTime="2.688058889s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 11:40:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:40:52.482048087 +0000 UTC m=+306.874985875" watchObservedRunningTime="2026-07-23 11:40:52.482841338 +0000 UTC m=+306.875779117"
Jul 23 11:40:52.972244 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:40:52.972203    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:40:53.104720 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:40:53.104643    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:40:55.284157 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:40:55.284122    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:40:55.467506 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:40:55.467477    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:40:57.037727 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:40:57.037662    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:40:57.098414 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:40:57.098380    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:41:00.247439 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:41:00.247395    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-x2g9r"]
Jul 23 11:41:00.249888 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:41:00.249863    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-x2g9r"
Jul 23 11:41:00.252234 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:41:00.252207    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Jul 23 11:41:00.252462 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:41:00.252446    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-nhnz6\""
Jul 23 11:41:00.253118 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:41:00.253103    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Jul 23 11:41:00.262521 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:41:00.262493    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-x2g9r"]
Jul 23 11:41:00.292533 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:41:00.292501    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7b62ed41-9d06-4ca4-b0ab-009a1149e9af-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-x2g9r\" (UID: \"7b62ed41-9d06-4ca4-b0ab-009a1149e9af\") " pod="cert-manager/cert-manager-webhook-6486ffd997-x2g9r"
Jul 23 11:41:00.292753 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:41:00.292550    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hkdwt\" (UniqueName: \"kubernetes.io/projected/7b62ed41-9d06-4ca4-b0ab-009a1149e9af-kube-api-access-hkdwt\") pod \"cert-manager-webhook-6486ffd997-x2g9r\" (UID: \"7b62ed41-9d06-4ca4-b0ab-009a1149e9af\") " pod="cert-manager/cert-manager-webhook-6486ffd997-x2g9r"
Jul 23 11:41:00.393473 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:41:00.393435    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hkdwt\" (UniqueName: \"kubernetes.io/projected/7b62ed41-9d06-4ca4-b0ab-009a1149e9af-kube-api-access-hkdwt\") pod \"cert-manager-webhook-6486ffd997-x2g9r\" (UID: \"7b62ed41-9d06-4ca4-b0ab-009a1149e9af\") " pod="cert-manager/cert-manager-webhook-6486ffd997-x2g9r"
Jul 23 11:41:00.393662 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:41:00.393517    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7b62ed41-9d06-4ca4-b0ab-009a1149e9af-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-x2g9r\" (UID: \"7b62ed41-9d06-4ca4-b0ab-009a1149e9af\") " pod="cert-manager/cert-manager-webhook-6486ffd997-x2g9r"
Jul 23 11:41:00.401667 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:41:00.401627    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7b62ed41-9d06-4ca4-b0ab-009a1149e9af-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-x2g9r\" (UID: \"7b62ed41-9d06-4ca4-b0ab-009a1149e9af\") " pod="cert-manager/cert-manager-webhook-6486ffd997-x2g9r"
Jul 23 11:41:00.401854 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:41:00.401835    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hkdwt\" (UniqueName: \"kubernetes.io/projected/7b62ed41-9d06-4ca4-b0ab-009a1149e9af-kube-api-access-hkdwt\") pod \"cert-manager-webhook-6486ffd997-x2g9r\" (UID: \"7b62ed41-9d06-4ca4-b0ab-009a1149e9af\") " pod="cert-manager/cert-manager-webhook-6486ffd997-x2g9r"
Jul 23 11:41:00.576908 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:41:00.576805    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-x2g9r"
Jul 23 11:41:00.718092 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:41:00.718057    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-x2g9r"]
Jul 23 11:41:00.719639 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:41:00.719606    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7b62ed41_9d06_4ca4_b0ab_009a1149e9af.slice/crio-0187e400655ca74699d129075ae9ff323d5631d68125ad141c3914e4de98b98a WatchSource:0}: Error finding container 0187e400655ca74699d129075ae9ff323d5631d68125ad141c3914e4de98b98a: Status 404 returned error can't find the container with id 0187e400655ca74699d129075ae9ff323d5631d68125ad141c3914e4de98b98a
Jul 23 11:41:01.489233 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:41:01.489191    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-x2g9r" event={"ID":"7b62ed41-9d06-4ca4-b0ab-009a1149e9af","Type":"ContainerStarted","Data":"0187e400655ca74699d129075ae9ff323d5631d68125ad141c3914e4de98b98a"}
Jul 23 11:41:02.677222 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:41:02.677177    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:41:03.744935 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:41:03.744842    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:41:03.810902 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:41:03.810855    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:41:04.504983 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:41:04.504944    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-x2g9r" event={"ID":"7b62ed41-9d06-4ca4-b0ab-009a1149e9af","Type":"ContainerStarted","Data":"a49765aa30e5d004add2cf8c1f38d986a96696259bf767b0cf8fee11b186e86c"}
Jul 23 11:41:04.505182 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:41:04.505022    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-x2g9r"
Jul 23 11:41:04.523448 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:41:04.523386    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-x2g9r" podStartSLOduration=1.763177411 podStartE2EDuration="4.523369137s" totalImagesPullingTime="2.760191726s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 11:41:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:41:04.522281699 +0000 UTC m=+318.915219487" watchObservedRunningTime="2026-07-23 11:41:04.523369137 +0000 UTC m=+318.916306924"
Jul 23 11:41:05.246939 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:41:05.246894    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:41:05.347076 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:41:05.347044    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:41:10.512357 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:41:10.512324    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-x2g9r"
Jul 23 11:41:10.734292 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:41:10.734251    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:41:10.832003 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:41:10.831922    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:41:12.561287 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:41:12.561253    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:41:27.111796 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:41:27.111742    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:41:27.205414 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:41:27.205380    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-544c65c659-4msr2"]
Jul 23 11:41:27.210511 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:41:27.210485    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-544c65c659-4msr2"
Jul 23 11:41:27.213916 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:41:27.213889    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Jul 23 11:41:27.214630 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:41:27.214610    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Jul 23 11:41:27.214727 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:41:27.214633    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Jul 23 11:41:27.214727 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:41:27.214618    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Jul 23 11:41:27.214825 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:41:27.214738    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Jul 23 11:41:27.214825 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:41:27.214803    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-r8vzg\""
Jul 23 11:41:27.239607 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:41:27.239573    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-544c65c659-4msr2"]
Jul 23 11:41:27.254091 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:41:27.254035    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:41:27.332706 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:41:27.332650    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/a558bac7-2851-4f03-ae33-76c40027ad4e-metrics-cert\") pod \"lws-controller-manager-544c65c659-4msr2\" (UID: \"a558bac7-2851-4f03-ae33-76c40027ad4e\") " pod="openshift-lws-operator/lws-controller-manager-544c65c659-4msr2"
Jul 23 11:41:27.332706 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:41:27.332702    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a558bac7-2851-4f03-ae33-76c40027ad4e-cert\") pod \"lws-controller-manager-544c65c659-4msr2\" (UID: \"a558bac7-2851-4f03-ae33-76c40027ad4e\") " pod="openshift-lws-operator/lws-controller-manager-544c65c659-4msr2"
Jul 23 11:41:27.332922 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:41:27.332735    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kzlh2\" (UniqueName: \"kubernetes.io/projected/a558bac7-2851-4f03-ae33-76c40027ad4e-kube-api-access-kzlh2\") pod \"lws-controller-manager-544c65c659-4msr2\" (UID: \"a558bac7-2851-4f03-ae33-76c40027ad4e\") " pod="openshift-lws-operator/lws-controller-manager-544c65c659-4msr2"
Jul 23 11:41:27.332922 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:41:27.332763    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/a558bac7-2851-4f03-ae33-76c40027ad4e-manager-config\") pod \"lws-controller-manager-544c65c659-4msr2\" (UID: \"a558bac7-2851-4f03-ae33-76c40027ad4e\") " pod="openshift-lws-operator/lws-controller-manager-544c65c659-4msr2"
Jul 23 11:41:27.434339 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:41:27.434292    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/a558bac7-2851-4f03-ae33-76c40027ad4e-metrics-cert\") pod \"lws-controller-manager-544c65c659-4msr2\" (UID: \"a558bac7-2851-4f03-ae33-76c40027ad4e\") " pod="openshift-lws-operator/lws-controller-manager-544c65c659-4msr2"
Jul 23 11:41:27.434339 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:41:27.434338    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a558bac7-2851-4f03-ae33-76c40027ad4e-cert\") pod \"lws-controller-manager-544c65c659-4msr2\" (UID: \"a558bac7-2851-4f03-ae33-76c40027ad4e\") " pod="openshift-lws-operator/lws-controller-manager-544c65c659-4msr2"
Jul 23 11:41:27.434676 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:41:27.434364    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzlh2\" (UniqueName: \"kubernetes.io/projected/a558bac7-2851-4f03-ae33-76c40027ad4e-kube-api-access-kzlh2\") pod \"lws-controller-manager-544c65c659-4msr2\" (UID: \"a558bac7-2851-4f03-ae33-76c40027ad4e\") " pod="openshift-lws-operator/lws-controller-manager-544c65c659-4msr2"
Jul 23 11:41:27.434676 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:41:27.434580    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/a558bac7-2851-4f03-ae33-76c40027ad4e-manager-config\") pod \"lws-controller-manager-544c65c659-4msr2\" (UID: \"a558bac7-2851-4f03-ae33-76c40027ad4e\") " pod="openshift-lws-operator/lws-controller-manager-544c65c659-4msr2"
Jul 23 11:41:27.435206 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:41:27.435178    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/a558bac7-2851-4f03-ae33-76c40027ad4e-manager-config\") pod \"lws-controller-manager-544c65c659-4msr2\" (UID: \"a558bac7-2851-4f03-ae33-76c40027ad4e\") " pod="openshift-lws-operator/lws-controller-manager-544c65c659-4msr2"
Jul 23 11:41:27.437023 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:41:27.437003    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a558bac7-2851-4f03-ae33-76c40027ad4e-cert\") pod \"lws-controller-manager-544c65c659-4msr2\" (UID: \"a558bac7-2851-4f03-ae33-76c40027ad4e\") " pod="openshift-lws-operator/lws-controller-manager-544c65c659-4msr2"
Jul 23 11:41:27.437134 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:41:27.437027    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/a558bac7-2851-4f03-ae33-76c40027ad4e-metrics-cert\") pod \"lws-controller-manager-544c65c659-4msr2\" (UID: \"a558bac7-2851-4f03-ae33-76c40027ad4e\") " pod="openshift-lws-operator/lws-controller-manager-544c65c659-4msr2"
Jul 23 11:41:27.444668 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:41:27.444642    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzlh2\" (UniqueName: \"kubernetes.io/projected/a558bac7-2851-4f03-ae33-76c40027ad4e-kube-api-access-kzlh2\") pod \"lws-controller-manager-544c65c659-4msr2\" (UID: \"a558bac7-2851-4f03-ae33-76c40027ad4e\") " pod="openshift-lws-operator/lws-controller-manager-544c65c659-4msr2"
Jul 23 11:41:27.520473 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:41:27.520426    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-544c65c659-4msr2"
Jul 23 11:41:27.677281 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:41:27.677247    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-544c65c659-4msr2"]
Jul 23 11:41:27.679163 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:41:27.679124    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda558bac7_2851_4f03_ae33_76c40027ad4e.slice/crio-1ed4302859de2ca0494e7aa2590b6684e37b6855880e71207a157a21efdda42d WatchSource:0}: Error finding container 1ed4302859de2ca0494e7aa2590b6684e37b6855880e71207a157a21efdda42d: Status 404 returned error can't find the container with id 1ed4302859de2ca0494e7aa2590b6684e37b6855880e71207a157a21efdda42d
Jul 23 11:41:28.609712 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:41:28.609654    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-544c65c659-4msr2" event={"ID":"a558bac7-2851-4f03-ae33-76c40027ad4e","Type":"ContainerStarted","Data":"1ed4302859de2ca0494e7aa2590b6684e37b6855880e71207a157a21efdda42d"}
Jul 23 11:41:29.065479 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:41:29.065439    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:41:29.204105 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:41:29.204013    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:41:29.709137 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:41:29.709101    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:41:30.227131 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:41:30.227037    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:41:30.590739 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:41:30.590627    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7c7646c6cc-qj2nv"]
Jul 23 11:41:30.595224 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:41:30.595173    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7c7646c6cc-qj2nv"
Jul 23 11:41:30.603182 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:41:30.602994    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Jul 23 11:41:30.603182 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:41:30.603075    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Jul 23 11:41:30.603182 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:41:30.603141    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-rwjxf\""
Jul 23 11:41:30.605145 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:41:30.605119    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Jul 23 11:41:30.614206 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:41:30.614180    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Jul 23 11:41:30.627805 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:41:30.627775    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:41:30.635210 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:41:30.635178    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7c7646c6cc-qj2nv"]
Jul 23 11:41:30.664867 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:41:30.664827    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/feaa1c4b-4dad-47ea-9511-95f0d907dc27-tmp\") pod \"opendatahub-operator-controller-manager-7c7646c6cc-qj2nv\" (UID: \"feaa1c4b-4dad-47ea-9511-95f0d907dc27\") " pod="opendatahub/opendatahub-operator-controller-manager-7c7646c6cc-qj2nv"
Jul 23 11:41:30.665023 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:41:30.664892    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/feaa1c4b-4dad-47ea-9511-95f0d907dc27-manifests\") pod \"opendatahub-operator-controller-manager-7c7646c6cc-qj2nv\" (UID: \"feaa1c4b-4dad-47ea-9511-95f0d907dc27\") " pod="opendatahub/opendatahub-operator-controller-manager-7c7646c6cc-qj2nv"
Jul 23 11:41:30.665023 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:41:30.664921    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/feaa1c4b-4dad-47ea-9511-95f0d907dc27-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7c7646c6cc-qj2nv\" (UID: \"feaa1c4b-4dad-47ea-9511-95f0d907dc27\") " pod="opendatahub/opendatahub-operator-controller-manager-7c7646c6cc-qj2nv"
Jul 23 11:41:30.665023 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:41:30.664973    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njm6f\" (UniqueName: \"kubernetes.io/projected/feaa1c4b-4dad-47ea-9511-95f0d907dc27-kube-api-access-njm6f\") pod \"opendatahub-operator-controller-manager-7c7646c6cc-qj2nv\" (UID: \"feaa1c4b-4dad-47ea-9511-95f0d907dc27\") " pod="opendatahub/opendatahub-operator-controller-manager-7c7646c6cc-qj2nv"
Jul 23 11:41:30.665023 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:41:30.665005    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/feaa1c4b-4dad-47ea-9511-95f0d907dc27-webhook-cert\") pod \"opendatahub-operator-controller-manager-7c7646c6cc-qj2nv\" (UID: \"feaa1c4b-4dad-47ea-9511-95f0d907dc27\") " pod="opendatahub/opendatahub-operator-controller-manager-7c7646c6cc-qj2nv"
Jul 23 11:41:30.765423 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:41:30.765398    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/feaa1c4b-4dad-47ea-9511-95f0d907dc27-tmp\") pod \"opendatahub-operator-controller-manager-7c7646c6cc-qj2nv\" (UID: \"feaa1c4b-4dad-47ea-9511-95f0d907dc27\") " pod="opendatahub/opendatahub-operator-controller-manager-7c7646c6cc-qj2nv"
Jul 23 11:41:30.765825 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:41:30.765453    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/feaa1c4b-4dad-47ea-9511-95f0d907dc27-manifests\") pod \"opendatahub-operator-controller-manager-7c7646c6cc-qj2nv\" (UID: \"feaa1c4b-4dad-47ea-9511-95f0d907dc27\") " pod="opendatahub/opendatahub-operator-controller-manager-7c7646c6cc-qj2nv"
Jul 23 11:41:30.765825 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:41:30.765484    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/feaa1c4b-4dad-47ea-9511-95f0d907dc27-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7c7646c6cc-qj2nv\" (UID: \"feaa1c4b-4dad-47ea-9511-95f0d907dc27\") " pod="opendatahub/opendatahub-operator-controller-manager-7c7646c6cc-qj2nv"
Jul 23 11:41:30.765825 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:41:30.765519    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njm6f\" (UniqueName: \"kubernetes.io/projected/feaa1c4b-4dad-47ea-9511-95f0d907dc27-kube-api-access-njm6f\") pod \"opendatahub-operator-controller-manager-7c7646c6cc-qj2nv\" (UID: \"feaa1c4b-4dad-47ea-9511-95f0d907dc27\") " pod="opendatahub/opendatahub-operator-controller-manager-7c7646c6cc-qj2nv"
Jul 23 11:41:30.765825 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:41:30.765558    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/feaa1c4b-4dad-47ea-9511-95f0d907dc27-webhook-cert\") pod \"opendatahub-operator-controller-manager-7c7646c6cc-qj2nv\" (UID: \"feaa1c4b-4dad-47ea-9511-95f0d907dc27\") " pod="opendatahub/opendatahub-operator-controller-manager-7c7646c6cc-qj2nv"
Jul 23 11:41:30.766041 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:41:30.765933    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/feaa1c4b-4dad-47ea-9511-95f0d907dc27-manifests\") pod \"opendatahub-operator-controller-manager-7c7646c6cc-qj2nv\" (UID: \"feaa1c4b-4dad-47ea-9511-95f0d907dc27\") " pod="opendatahub/opendatahub-operator-controller-manager-7c7646c6cc-qj2nv"
Jul 23 11:41:30.767779 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:41:30.767750    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/feaa1c4b-4dad-47ea-9511-95f0d907dc27-tmp\") pod \"opendatahub-operator-controller-manager-7c7646c6cc-qj2nv\" (UID: \"feaa1c4b-4dad-47ea-9511-95f0d907dc27\") " pod="opendatahub/opendatahub-operator-controller-manager-7c7646c6cc-qj2nv"
Jul 23 11:41:30.768105 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:41:30.768082    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/feaa1c4b-4dad-47ea-9511-95f0d907dc27-webhook-cert\") pod \"opendatahub-operator-controller-manager-7c7646c6cc-qj2nv\" (UID: \"feaa1c4b-4dad-47ea-9511-95f0d907dc27\") " pod="opendatahub/opendatahub-operator-controller-manager-7c7646c6cc-qj2nv"
Jul 23 11:41:30.768173 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:41:30.768143    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/feaa1c4b-4dad-47ea-9511-95f0d907dc27-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7c7646c6cc-qj2nv\" (UID: \"feaa1c4b-4dad-47ea-9511-95f0d907dc27\") " pod="opendatahub/opendatahub-operator-controller-manager-7c7646c6cc-qj2nv"
Jul 23 11:41:30.800301 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:41:30.800271    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-njm6f\" (UniqueName: \"kubernetes.io/projected/feaa1c4b-4dad-47ea-9511-95f0d907dc27-kube-api-access-njm6f\") pod \"opendatahub-operator-controller-manager-7c7646c6cc-qj2nv\" (UID: \"feaa1c4b-4dad-47ea-9511-95f0d907dc27\") " pod="opendatahub/opendatahub-operator-controller-manager-7c7646c6cc-qj2nv"
Jul 23 11:41:30.911194 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:41:30.911156    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7c7646c6cc-qj2nv"
Jul 23 11:41:31.069265 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:41:31.069237    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7c7646c6cc-qj2nv"]
Jul 23 11:41:31.072318 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:41:31.072285    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfeaa1c4b_4dad_47ea_9511_95f0d907dc27.slice/crio-5f6bfc26131245d4dede2adc9961e534aab2328d3a990d9fbe8a8d1e72b27825 WatchSource:0}: Error finding container 5f6bfc26131245d4dede2adc9961e534aab2328d3a990d9fbe8a8d1e72b27825: Status 404 returned error can't find the container with id 5f6bfc26131245d4dede2adc9961e534aab2328d3a990d9fbe8a8d1e72b27825
Jul 23 11:41:31.199108 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:41:31.199016    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:41:31.333726 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:41:31.333665    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:41:31.562716 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:41:31.562597    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:41:31.624373 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:41:31.624334    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7c7646c6cc-qj2nv" event={"ID":"feaa1c4b-4dad-47ea-9511-95f0d907dc27","Type":"ContainerStarted","Data":"5f6bfc26131245d4dede2adc9961e534aab2328d3a990d9fbe8a8d1e72b27825"}
Jul 23 11:41:31.625787 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:41:31.625755    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-544c65c659-4msr2" event={"ID":"a558bac7-2851-4f03-ae33-76c40027ad4e","Type":"ContainerStarted","Data":"d6a5553d7e2d29d75c05ac3f591b95245045a5872ebad29cf58d6638c439961e"}
Jul 23 11:41:31.625916 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:41:31.625885    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-544c65c659-4msr2"
Jul 23 11:41:31.666937 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:41:31.666883    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-544c65c659-4msr2" podStartSLOduration=1.6491922140000002 podStartE2EDuration="4.666867505s" totalImagesPullingTime="3.017675291s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 11:41:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:41:31.665569727 +0000 UTC m=+346.058507515" watchObservedRunningTime="2026-07-23 11:41:31.666867505 +0000 UTC m=+346.059805292"
Jul 23 11:41:31.732281 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:41:31.732245    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:41:31.881345 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:41:31.881317    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:41:34.641898 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:41:34.641865    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7c7646c6cc-qj2nv" event={"ID":"feaa1c4b-4dad-47ea-9511-95f0d907dc27","Type":"ContainerStarted","Data":"dd072f00d281749ca38078c09b89f6338b3ed3aabb071733cdfede1c2026244e"}
Jul 23 11:41:35.648157 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:41:35.648065    2563 generic.go:356] "Generic (PLEG): container finished" podID="feaa1c4b-4dad-47ea-9511-95f0d907dc27" containerID="dd072f00d281749ca38078c09b89f6338b3ed3aabb071733cdfede1c2026244e" exitCode=0
Jul 23 11:41:35.648507 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:41:35.648155    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7c7646c6cc-qj2nv" event={"ID":"feaa1c4b-4dad-47ea-9511-95f0d907dc27","Type":"ContainerDied","Data":"dd072f00d281749ca38078c09b89f6338b3ed3aabb071733cdfede1c2026244e"}
Jul 23 11:41:36.654740 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:41:36.654631    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7c7646c6cc-qj2nv" event={"ID":"feaa1c4b-4dad-47ea-9511-95f0d907dc27","Type":"ContainerStarted","Data":"7b2ea8c2c994739a06c91d47d5a7638c948c7e76e317a86a877e03dedd215c86"}
Jul 23 11:41:36.655069 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:41:36.654845    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-7c7646c6cc-qj2nv"
Jul 23 11:41:36.688675 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:41:36.688621    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-7c7646c6cc-qj2nv" podStartSLOduration=2.020556459 podStartE2EDuration="6.688604871s" totalImagesPullingTime="3.624848872s" totalInitContainerRuntime="1.04319954s" isStatefulPod=false podCreationTimestamp="2026-07-23 11:41:30 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:41:36.68708221 +0000 UTC m=+351.080019997" watchObservedRunningTime="2026-07-23 11:41:36.688604871 +0000 UTC m=+351.081542658"
Jul 23 11:41:41.873637 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:41:41.873600    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:41:41.988016 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:41:41.987982    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:41:42.634105 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:41:42.634070    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-544c65c659-4msr2"
Jul 23 11:41:43.873996 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:41:43.873966    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:41:47.661953 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:41:47.661923    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-7c7646c6cc-qj2nv"
Jul 23 11:41:50.833980 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:41:50.833937    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:41:51.007000 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:41:51.006966    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:41:51.405887 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:41:51.405853    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:41:51.432157 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:41:51.432124    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-5b7f786dc8-5gcr8"]
Jul 23 11:41:51.440765 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:41:51.440743    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5b7f786dc8-5gcr8"
Jul 23 11:41:51.448260 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:41:51.448239    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Jul 23 11:41:51.448384 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:41:51.448317    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Jul 23 11:41:51.448471 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:41:51.448445    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 23 11:41:51.448529 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:41:51.448465    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 23 11:41:51.448529 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:41:51.448511    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-jlcm2\""
Jul 23 11:41:51.454119 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:41:51.454098    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5b7f786dc8-5gcr8"]
Jul 23 11:41:51.543800 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:41:51.543773    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f7bb354b-f7b9-4a34-a239-4e985a64ce1c-tmp\") pod \"kube-auth-proxy-5b7f786dc8-5gcr8\" (UID: \"f7bb354b-f7b9-4a34-a239-4e985a64ce1c\") " pod="openshift-ingress/kube-auth-proxy-5b7f786dc8-5gcr8"
Jul 23 11:41:51.543960 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:41:51.543839    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f7bb354b-f7b9-4a34-a239-4e985a64ce1c-tls-certs\") pod \"kube-auth-proxy-5b7f786dc8-5gcr8\" (UID: \"f7bb354b-f7b9-4a34-a239-4e985a64ce1c\") " pod="openshift-ingress/kube-auth-proxy-5b7f786dc8-5gcr8"
Jul 23 11:41:51.543960 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:41:51.543864    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7r2r\" (UniqueName: \"kubernetes.io/projected/f7bb354b-f7b9-4a34-a239-4e985a64ce1c-kube-api-access-q7r2r\") pod \"kube-auth-proxy-5b7f786dc8-5gcr8\" (UID: \"f7bb354b-f7b9-4a34-a239-4e985a64ce1c\") " pod="openshift-ingress/kube-auth-proxy-5b7f786dc8-5gcr8"
Jul 23 11:41:51.645062 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:41:51.645011    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f7bb354b-f7b9-4a34-a239-4e985a64ce1c-tls-certs\") pod \"kube-auth-proxy-5b7f786dc8-5gcr8\" (UID: \"f7bb354b-f7b9-4a34-a239-4e985a64ce1c\") " pod="openshift-ingress/kube-auth-proxy-5b7f786dc8-5gcr8"
Jul 23 11:41:51.645231 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:41:51.645081    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7r2r\" (UniqueName: \"kubernetes.io/projected/f7bb354b-f7b9-4a34-a239-4e985a64ce1c-kube-api-access-q7r2r\") pod \"kube-auth-proxy-5b7f786dc8-5gcr8\" (UID: \"f7bb354b-f7b9-4a34-a239-4e985a64ce1c\") " pod="openshift-ingress/kube-auth-proxy-5b7f786dc8-5gcr8"
Jul 23 11:41:51.645231 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:41:51.645129    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f7bb354b-f7b9-4a34-a239-4e985a64ce1c-tmp\") pod \"kube-auth-proxy-5b7f786dc8-5gcr8\" (UID: \"f7bb354b-f7b9-4a34-a239-4e985a64ce1c\") " pod="openshift-ingress/kube-auth-proxy-5b7f786dc8-5gcr8"
Jul 23 11:41:51.647464 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:41:51.647436    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f7bb354b-f7b9-4a34-a239-4e985a64ce1c-tmp\") pod \"kube-auth-proxy-5b7f786dc8-5gcr8\" (UID: \"f7bb354b-f7b9-4a34-a239-4e985a64ce1c\") " pod="openshift-ingress/kube-auth-proxy-5b7f786dc8-5gcr8"
Jul 23 11:41:51.647738 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:41:51.647720    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f7bb354b-f7b9-4a34-a239-4e985a64ce1c-tls-certs\") pod \"kube-auth-proxy-5b7f786dc8-5gcr8\" (UID: \"f7bb354b-f7b9-4a34-a239-4e985a64ce1c\") " pod="openshift-ingress/kube-auth-proxy-5b7f786dc8-5gcr8"
Jul 23 11:41:51.668926 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:41:51.668850    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:41:51.673997 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:41:51.673973    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7r2r\" (UniqueName: \"kubernetes.io/projected/f7bb354b-f7b9-4a34-a239-4e985a64ce1c-kube-api-access-q7r2r\") pod \"kube-auth-proxy-5b7f786dc8-5gcr8\" (UID: \"f7bb354b-f7b9-4a34-a239-4e985a64ce1c\") " pod="openshift-ingress/kube-auth-proxy-5b7f786dc8-5gcr8"
Jul 23 11:41:51.749862 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:41:51.749827    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5b7f786dc8-5gcr8"
Jul 23 11:41:51.853655 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:41:51.853625    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:41:51.909904 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:41:51.909865    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5b7f786dc8-5gcr8"]
Jul 23 11:41:51.915525 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:41:51.915498    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7bb354b_f7b9_4a34_a239_4e985a64ce1c.slice/crio-c420b062433cf62973a84e4055182e62e3fcfbb3ddcb9f6ece779969fb5d25e4 WatchSource:0}: Error finding container c420b062433cf62973a84e4055182e62e3fcfbb3ddcb9f6ece779969fb5d25e4: Status 404 returned error can't find the container with id c420b062433cf62973a84e4055182e62e3fcfbb3ddcb9f6ece779969fb5d25e4
Jul 23 11:41:52.014984 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:41:52.014950    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:41:52.302713 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:41:52.302609    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:41:52.678857 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:41:52.678824    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:41:52.720308 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:41:52.720274    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5b7f786dc8-5gcr8" event={"ID":"f7bb354b-f7b9-4a34-a239-4e985a64ce1c","Type":"ContainerStarted","Data":"c420b062433cf62973a84e4055182e62e3fcfbb3ddcb9f6ece779969fb5d25e4"}
Jul 23 11:41:52.866923 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:41:52.866883    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:41:53.638958 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:41:53.638923    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:41:53.740632 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:41:53.740570    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:41:54.729977 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:41:54.729945    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5b7f786dc8-5gcr8" event={"ID":"f7bb354b-f7b9-4a34-a239-4e985a64ce1c","Type":"ContainerStarted","Data":"8e1ba5acf39f0a85de940ca209ada7c38dd1d7c795fdab9b27e6594ac85412ba"}
Jul 23 11:41:54.749861 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:41:54.749799    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-5b7f786dc8-5gcr8" podStartSLOduration=1.148963714 podStartE2EDuration="3.749784147s" totalImagesPullingTime="2.600820433s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 11:41:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:41:54.748382289 +0000 UTC m=+369.141320146" watchObservedRunningTime="2026-07-23 11:41:54.749784147 +0000 UTC m=+369.142721934"
Jul 23 11:41:54.965087 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:41:54.965047    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:42:02.365453 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:42:02.365418    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:42:02.520523 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:42:02.520486    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:42:02.639071 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:42:02.638979    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:42:02.754614 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:42:02.754574    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:42:03.767234 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:42:03.767190    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:42:03.958175 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:42:03.958141    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:42:04.112994 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:42:04.112908    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:42:05.267804 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:42:05.267768    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:42:05.669038 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:42:05.668995    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:42:05.794904 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:42:05.794871    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:42:13.653284 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:42:13.653247    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:42:13.866018 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:42:13.865987    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:42:47.163979 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:42:47.163926    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:42:52.359635 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:42:52.359595    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:42:52.459089 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:42:52.459052    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:42:52.752716 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:42:52.752667    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:43:21.293816 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:43:21.293713    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:43:21.898828 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:43:21.898788    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:43:22.921365 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:43:22.921326    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:43:23.019166 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:43:23.019121    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:43:26.461272 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:43:26.461240    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:43:29.314120 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:43:29.314088    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:43:29.951546 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:43:29.951509    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:43:31.685895 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:43:31.685863    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:43:32.301367 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:43:32.301330    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:43:33.044556 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:43:33.044514    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:43:33.143721 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:43:33.143663    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:43:33.241159 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:43:33.241115    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:43:33.898212 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:43:33.898174    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:43:33.995141 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:43:33.995102    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:43:35.183627 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:43:35.183596    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:43:36.858350 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:43:36.858301    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:43:36.958520 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:43:36.958492    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:43:37.270106 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:43:37.270062    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:43:37.366819 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:43:37.366784    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:43:40.661798 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:43:40.661747    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:43:40.737524 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:43:40.737486    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:43:43.119142 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:43:43.119101    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-gxbsp"]
Jul 23 11:43:43.122372 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:43:43.122348    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-gxbsp"
Jul 23 11:43:43.124772 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:43:43.124739    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Jul 23 11:43:43.124937 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:43:43.124827    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 23 11:43:43.124937 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:43:43.124922    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Jul 23 11:43:43.125663 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:43:43.125633    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 23 11:43:43.125663 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:43:43.125648    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-gqsgb\""
Jul 23 11:43:43.125924 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:43:43.125902    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:43:43.130587 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:43:43.130563    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-gxbsp"]
Jul 23 11:43:43.192018 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:43:43.191968    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwmlq\" (UniqueName: \"kubernetes.io/projected/be20ca05-298e-47c1-9e10-e36c4200b713-kube-api-access-jwmlq\") pod \"kuadrant-console-plugin-66b5774985-gxbsp\" (UID: \"be20ca05-298e-47c1-9e10-e36c4200b713\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-gxbsp"
Jul 23 11:43:43.192230 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:43:43.192027    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/be20ca05-298e-47c1-9e10-e36c4200b713-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-gxbsp\" (UID: \"be20ca05-298e-47c1-9e10-e36c4200b713\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-gxbsp"
Jul 23 11:43:43.192230 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:43:43.192094    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/be20ca05-298e-47c1-9e10-e36c4200b713-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-gxbsp\" (UID: \"be20ca05-298e-47c1-9e10-e36c4200b713\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-gxbsp"
Jul 23 11:43:43.292958 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:43:43.292913    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwmlq\" (UniqueName: \"kubernetes.io/projected/be20ca05-298e-47c1-9e10-e36c4200b713-kube-api-access-jwmlq\") pod \"kuadrant-console-plugin-66b5774985-gxbsp\" (UID: \"be20ca05-298e-47c1-9e10-e36c4200b713\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-gxbsp"
Jul 23 11:43:43.293149 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:43:43.292988    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/be20ca05-298e-47c1-9e10-e36c4200b713-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-gxbsp\" (UID: \"be20ca05-298e-47c1-9e10-e36c4200b713\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-gxbsp"
Jul 23 11:43:43.293149 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:43:43.293071    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/be20ca05-298e-47c1-9e10-e36c4200b713-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-gxbsp\" (UID: \"be20ca05-298e-47c1-9e10-e36c4200b713\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-gxbsp"
Jul 23 11:43:43.293227 ip-10-0-141-229 kubenswrapper[2563]: E0723 11:43:43.293155    2563 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found
Jul 23 11:43:43.293260 ip-10-0-141-229 kubenswrapper[2563]: E0723 11:43:43.293231    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be20ca05-298e-47c1-9e10-e36c4200b713-plugin-serving-cert podName:be20ca05-298e-47c1-9e10-e36c4200b713 nodeName:}" failed. No retries permitted until 2026-07-23 11:43:43.793213552 +0000 UTC m=+478.186151320 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/be20ca05-298e-47c1-9e10-e36c4200b713-plugin-serving-cert") pod "kuadrant-console-plugin-66b5774985-gxbsp" (UID: "be20ca05-298e-47c1-9e10-e36c4200b713") : secret "plugin-serving-cert" not found
Jul 23 11:43:43.293748 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:43:43.293677    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/be20ca05-298e-47c1-9e10-e36c4200b713-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-gxbsp\" (UID: \"be20ca05-298e-47c1-9e10-e36c4200b713\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-gxbsp"
Jul 23 11:43:43.306148 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:43:43.306119    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwmlq\" (UniqueName: \"kubernetes.io/projected/be20ca05-298e-47c1-9e10-e36c4200b713-kube-api-access-jwmlq\") pod \"kuadrant-console-plugin-66b5774985-gxbsp\" (UID: \"be20ca05-298e-47c1-9e10-e36c4200b713\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-gxbsp"
Jul 23 11:43:43.639004 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:43:43.638963    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:43:43.796755 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:43:43.796714    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/be20ca05-298e-47c1-9e10-e36c4200b713-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-gxbsp\" (UID: \"be20ca05-298e-47c1-9e10-e36c4200b713\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-gxbsp"
Jul 23 11:43:43.799657 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:43:43.799617    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/be20ca05-298e-47c1-9e10-e36c4200b713-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-gxbsp\" (UID: \"be20ca05-298e-47c1-9e10-e36c4200b713\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-gxbsp"
Jul 23 11:43:43.879605 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:43:43.879567    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:43:43.979479 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:43:43.979381    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:43:44.033257 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:43:44.033215    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-gxbsp"
Jul 23 11:43:44.080281 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:43:44.080241    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:43:44.172851 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:43:44.172814    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-gxbsp"]
Jul 23 11:43:44.176078 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:43:44.176049    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbe20ca05_298e_47c1_9e10_e36c4200b713.slice/crio-3731a6432c78ce5ae20ee8096b2cfdf5141a8b90ebc80c6951357880cf7f2c09 WatchSource:0}: Error finding container 3731a6432c78ce5ae20ee8096b2cfdf5141a8b90ebc80c6951357880cf7f2c09: Status 404 returned error can't find the container with id 3731a6432c78ce5ae20ee8096b2cfdf5141a8b90ebc80c6951357880cf7f2c09
Jul 23 11:43:44.177317 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:43:44.177271    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:43:44.209607 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:43:44.209566    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-gxbsp" event={"ID":"be20ca05-298e-47c1-9e10-e36c4200b713","Type":"ContainerStarted","Data":"3731a6432c78ce5ae20ee8096b2cfdf5141a8b90ebc80c6951357880cf7f2c09"}
Jul 23 11:43:48.111141 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:43:48.111073    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:43:48.212159 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:43:48.212122    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:43:53.644668 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:43:53.644632    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-clvn8"]
Jul 23 11:43:53.672269 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:43:53.672235    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-clvn8"
Jul 23 11:43:53.677439 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:43:53.677407    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-clvn8"]
Jul 23 11:43:53.681209 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:43:53.680835    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-qddrk\""
Jul 23 11:43:53.792889 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:43:53.792847    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/88ca4a7c-d543-4ebf-9ebf-25257c44b69d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-clvn8\" (UID: \"88ca4a7c-d543-4ebf-9ebf-25257c44b69d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-clvn8"
Jul 23 11:43:53.793107 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:43:53.792915    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzld7\" (UniqueName: \"kubernetes.io/projected/88ca4a7c-d543-4ebf-9ebf-25257c44b69d-kube-api-access-rzld7\") pod \"kuadrant-operator-controller-manager-55467d9f7-clvn8\" (UID: \"88ca4a7c-d543-4ebf-9ebf-25257c44b69d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-clvn8"
Jul 23 11:43:53.894278 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:43:53.894225    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/88ca4a7c-d543-4ebf-9ebf-25257c44b69d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-clvn8\" (UID: \"88ca4a7c-d543-4ebf-9ebf-25257c44b69d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-clvn8"
Jul 23 11:43:53.894475 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:43:53.894292    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzld7\" (UniqueName: \"kubernetes.io/projected/88ca4a7c-d543-4ebf-9ebf-25257c44b69d-kube-api-access-rzld7\") pod \"kuadrant-operator-controller-manager-55467d9f7-clvn8\" (UID: \"88ca4a7c-d543-4ebf-9ebf-25257c44b69d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-clvn8"
Jul 23 11:43:53.894729 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:43:53.894661    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/88ca4a7c-d543-4ebf-9ebf-25257c44b69d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-clvn8\" (UID: \"88ca4a7c-d543-4ebf-9ebf-25257c44b69d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-clvn8"
Jul 23 11:43:53.915754 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:43:53.915721    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzld7\" (UniqueName: \"kubernetes.io/projected/88ca4a7c-d543-4ebf-9ebf-25257c44b69d-kube-api-access-rzld7\") pod \"kuadrant-operator-controller-manager-55467d9f7-clvn8\" (UID: \"88ca4a7c-d543-4ebf-9ebf-25257c44b69d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-clvn8"
Jul 23 11:43:53.986846 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:43:53.986807    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-clvn8"
Jul 23 11:43:56.143839 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:43:56.143780    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:43:56.247159 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:43:56.247120    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:43:56.784533 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:43:56.784487    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:43:56.882294 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:43:56.882261    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:44:00.192276 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:44:00.192231    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:44:00.290660 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:44:00.290595    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:44:02.348234 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:44:02.348141    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:44:02.448507 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:44:02.448442    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:44:06.065343 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:44:06.065318    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-clvn8"]
Jul 23 11:44:06.203664 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:44:06.203579    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88ca4a7c_d543_4ebf_9ebf_25257c44b69d.slice/crio-e457c290b58a9f404208e9f7e6ab180e202a9f558f20ef6409fadcd88e6f93f2 WatchSource:0}: Error finding container e457c290b58a9f404208e9f7e6ab180e202a9f558f20ef6409fadcd88e6f93f2: Status 404 returned error can't find the container with id e457c290b58a9f404208e9f7e6ab180e202a9f558f20ef6409fadcd88e6f93f2
Jul 23 11:44:06.323057 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:44:06.323018    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-clvn8" event={"ID":"88ca4a7c-d543-4ebf-9ebf-25257c44b69d","Type":"ContainerStarted","Data":"e457c290b58a9f404208e9f7e6ab180e202a9f558f20ef6409fadcd88e6f93f2"}
Jul 23 11:44:06.907287 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:44:06.907248    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:44:07.005343 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:44:07.005303    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:44:07.329362 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:44:07.329280    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-gxbsp" event={"ID":"be20ca05-298e-47c1-9e10-e36c4200b713","Type":"ContainerStarted","Data":"639fdd0a9dcfad150ee3b7c70f8f5227a997896797cb5e9fa19ad7008d32613b"}
Jul 23 11:44:07.347819 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:44:07.347760    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-gxbsp" podStartSLOduration=2.2777560660000002 podStartE2EDuration="24.347744123s" totalImagesPullingTime="22.069988057s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 11:43:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:44:07.346192521 +0000 UTC m=+501.739130309" watchObservedRunningTime="2026-07-23 11:44:07.347744123 +0000 UTC m=+501.740681910"
Jul 23 11:44:11.350625 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:44:11.350587    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-clvn8" event={"ID":"88ca4a7c-d543-4ebf-9ebf-25257c44b69d","Type":"ContainerStarted","Data":"6d944b667b26ddad9d546d77c7bce36afee5f61e64f9f634ee6d6c553e911fea"}
Jul 23 11:44:11.351063 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:44:11.350709    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-clvn8"
Jul 23 11:44:11.374416 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:44:11.374355    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-clvn8" podStartSLOduration=13.866626088 podStartE2EDuration="18.374339462s" totalImagesPullingTime="4.507713374s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 11:43:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:44:11.372802543 +0000 UTC m=+505.765740345" watchObservedRunningTime="2026-07-23 11:44:11.374339462 +0000 UTC m=+505.767277250"
Jul 23 11:44:22.357306 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:44:22.357269    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-clvn8"
Jul 23 11:44:34.997998 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:44:34.997965    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-clvn8"]
Jul 23 11:44:34.998447 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:44:34.998201    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-clvn8" podUID="88ca4a7c-d543-4ebf-9ebf-25257c44b69d" containerName="manager" containerID="cri-o://6d944b667b26ddad9d546d77c7bce36afee5f61e64f9f634ee6d6c553e911fea" gracePeriod=10
Jul 23 11:44:35.268066 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:44:35.267995    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-clvn8"
Jul 23 11:44:35.378177 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:44:35.378139    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rzld7\" (UniqueName: \"kubernetes.io/projected/88ca4a7c-d543-4ebf-9ebf-25257c44b69d-kube-api-access-rzld7\") pod \"88ca4a7c-d543-4ebf-9ebf-25257c44b69d\" (UID: \"88ca4a7c-d543-4ebf-9ebf-25257c44b69d\") "
Jul 23 11:44:35.378352 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:44:35.378191    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/88ca4a7c-d543-4ebf-9ebf-25257c44b69d-extensions-socket-volume\") pod \"88ca4a7c-d543-4ebf-9ebf-25257c44b69d\" (UID: \"88ca4a7c-d543-4ebf-9ebf-25257c44b69d\") "
Jul 23 11:44:35.378753 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:44:35.378726    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88ca4a7c-d543-4ebf-9ebf-25257c44b69d-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "88ca4a7c-d543-4ebf-9ebf-25257c44b69d" (UID: "88ca4a7c-d543-4ebf-9ebf-25257c44b69d"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 11:44:35.380557 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:44:35.380531    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88ca4a7c-d543-4ebf-9ebf-25257c44b69d-kube-api-access-rzld7" (OuterVolumeSpecName: "kube-api-access-rzld7") pod "88ca4a7c-d543-4ebf-9ebf-25257c44b69d" (UID: "88ca4a7c-d543-4ebf-9ebf-25257c44b69d"). InnerVolumeSpecName "kube-api-access-rzld7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 11:44:35.456611 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:44:35.456569    2563 generic.go:356] "Generic (PLEG): container finished" podID="88ca4a7c-d543-4ebf-9ebf-25257c44b69d" containerID="6d944b667b26ddad9d546d77c7bce36afee5f61e64f9f634ee6d6c553e911fea" exitCode=0
Jul 23 11:44:35.456819 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:44:35.456625    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-clvn8"
Jul 23 11:44:35.456819 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:44:35.456668    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-clvn8" event={"ID":"88ca4a7c-d543-4ebf-9ebf-25257c44b69d","Type":"ContainerDied","Data":"6d944b667b26ddad9d546d77c7bce36afee5f61e64f9f634ee6d6c553e911fea"}
Jul 23 11:44:35.456819 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:44:35.456752    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-clvn8" event={"ID":"88ca4a7c-d543-4ebf-9ebf-25257c44b69d","Type":"ContainerDied","Data":"e457c290b58a9f404208e9f7e6ab180e202a9f558f20ef6409fadcd88e6f93f2"}
Jul 23 11:44:35.456819 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:44:35.456784    2563 scope.go:117] "RemoveContainer" containerID="6d944b667b26ddad9d546d77c7bce36afee5f61e64f9f634ee6d6c553e911fea"
Jul 23 11:44:35.466049 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:44:35.466032    2563 scope.go:117] "RemoveContainer" containerID="6d944b667b26ddad9d546d77c7bce36afee5f61e64f9f634ee6d6c553e911fea"
Jul 23 11:44:35.466308 ip-10-0-141-229 kubenswrapper[2563]: E0723 11:44:35.466292    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d944b667b26ddad9d546d77c7bce36afee5f61e64f9f634ee6d6c553e911fea\": container with ID starting with 6d944b667b26ddad9d546d77c7bce36afee5f61e64f9f634ee6d6c553e911fea not found: ID does not exist" containerID="6d944b667b26ddad9d546d77c7bce36afee5f61e64f9f634ee6d6c553e911fea"
Jul 23 11:44:35.466389 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:44:35.466316    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d944b667b26ddad9d546d77c7bce36afee5f61e64f9f634ee6d6c553e911fea"} err="failed to get container status \"6d944b667b26ddad9d546d77c7bce36afee5f61e64f9f634ee6d6c553e911fea\": rpc error: code = NotFound desc = could not find container \"6d944b667b26ddad9d546d77c7bce36afee5f61e64f9f634ee6d6c553e911fea\": container with ID starting with 6d944b667b26ddad9d546d77c7bce36afee5f61e64f9f634ee6d6c553e911fea not found: ID does not exist"
Jul 23 11:44:35.478139 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:44:35.478115    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-clvn8"]
Jul 23 11:44:35.478893 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:44:35.478875    2563 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/88ca4a7c-d543-4ebf-9ebf-25257c44b69d-extensions-socket-volume\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 23 11:44:35.478955 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:44:35.478895    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rzld7\" (UniqueName: \"kubernetes.io/projected/88ca4a7c-d543-4ebf-9ebf-25257c44b69d-kube-api-access-rzld7\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 23 11:44:35.484155 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:44:35.484132    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-clvn8"]
Jul 23 11:44:36.077596 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:44:36.077555    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="88ca4a7c-d543-4ebf-9ebf-25257c44b69d" path="/var/lib/kubelet/pods/88ca4a7c-d543-4ebf-9ebf-25257c44b69d/volumes"
Jul 23 11:44:38.998361 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:44:38.998269    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:44:39.063640 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:44:39.063605    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:44:41.159438 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:44:41.159399    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:44:42.355396 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:44:42.355357    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-97hxc"]
Jul 23 11:44:42.355795 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:44:42.355672    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88ca4a7c-d543-4ebf-9ebf-25257c44b69d" containerName="manager"
Jul 23 11:44:42.355795 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:44:42.355697    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="88ca4a7c-d543-4ebf-9ebf-25257c44b69d" containerName="manager"
Jul 23 11:44:42.355795 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:44:42.355749    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="88ca4a7c-d543-4ebf-9ebf-25257c44b69d" containerName="manager"
Jul 23 11:44:42.358286 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:44:42.358269    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-97hxc"
Jul 23 11:44:42.359993 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:44:42.359961    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:44:42.360864 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:44:42.360843    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Jul 23 11:44:42.367337 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:44:42.367312    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-97hxc"]
Jul 23 11:44:42.451704 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:44:42.451656    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-97hxc"]
Jul 23 11:44:42.469147 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:44:42.469112    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:44:42.529180 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:44:42.529136    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:44:42.542328 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:44:42.542297    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ds2s\" (UniqueName: \"kubernetes.io/projected/cede1f68-63e6-4a1e-85d8-cbd8ac747220-kube-api-access-9ds2s\") pod \"limitador-limitador-64c94fd64f-97hxc\" (UID: \"cede1f68-63e6-4a1e-85d8-cbd8ac747220\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-97hxc"
Jul 23 11:44:42.542562 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:44:42.542361    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/cede1f68-63e6-4a1e-85d8-cbd8ac747220-config-file\") pod \"limitador-limitador-64c94fd64f-97hxc\" (UID: \"cede1f68-63e6-4a1e-85d8-cbd8ac747220\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-97hxc"
Jul 23 11:44:42.643083 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:44:42.643045    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/cede1f68-63e6-4a1e-85d8-cbd8ac747220-config-file\") pod \"limitador-limitador-64c94fd64f-97hxc\" (UID: \"cede1f68-63e6-4a1e-85d8-cbd8ac747220\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-97hxc"
Jul 23 11:44:42.643275 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:44:42.643123    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ds2s\" (UniqueName: \"kubernetes.io/projected/cede1f68-63e6-4a1e-85d8-cbd8ac747220-kube-api-access-9ds2s\") pod \"limitador-limitador-64c94fd64f-97hxc\" (UID: \"cede1f68-63e6-4a1e-85d8-cbd8ac747220\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-97hxc"
Jul 23 11:44:42.643664 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:44:42.643643    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/cede1f68-63e6-4a1e-85d8-cbd8ac747220-config-file\") pod \"limitador-limitador-64c94fd64f-97hxc\" (UID: \"cede1f68-63e6-4a1e-85d8-cbd8ac747220\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-97hxc"
Jul 23 11:44:42.662052 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:44:42.662023    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ds2s\" (UniqueName: \"kubernetes.io/projected/cede1f68-63e6-4a1e-85d8-cbd8ac747220-kube-api-access-9ds2s\") pod \"limitador-limitador-64c94fd64f-97hxc\" (UID: \"cede1f68-63e6-4a1e-85d8-cbd8ac747220\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-97hxc"
Jul 23 11:44:42.668885 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:44:42.668857    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-97hxc"
Jul 23 11:44:42.807339 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:44:42.807313    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-97hxc"]
Jul 23 11:44:42.809970 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:44:42.809944    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcede1f68_63e6_4a1e_85d8_cbd8ac747220.slice/crio-04331138ab910c1c5243d54da7c4c58e5389088148d8ead2fbfd98cdef8c180c WatchSource:0}: Error finding container 04331138ab910c1c5243d54da7c4c58e5389088148d8ead2fbfd98cdef8c180c: Status 404 returned error can't find the container with id 04331138ab910c1c5243d54da7c4c58e5389088148d8ead2fbfd98cdef8c180c
Jul 23 11:44:43.202224 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:44:43.202172    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:44:43.302523 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:44:43.302487    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:44:43.398348 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:44:43.398310    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-75d556bd77-cqspn"]
Jul 23 11:44:43.431097 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:44:43.431061    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-cqspn"]
Jul 23 11:44:43.431248 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:44:43.431163    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-cqspn"
Jul 23 11:44:43.433912 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:44:43.433893    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-248mc\""
Jul 23 11:44:43.493915 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:44:43.493830    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-97hxc" event={"ID":"cede1f68-63e6-4a1e-85d8-cbd8ac747220","Type":"ContainerStarted","Data":"04331138ab910c1c5243d54da7c4c58e5389088148d8ead2fbfd98cdef8c180c"}
Jul 23 11:44:43.551445 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:44:43.551407    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zz7ht\" (UniqueName: \"kubernetes.io/projected/29c00da5-0951-452a-bf66-cebd7f5a156b-kube-api-access-zz7ht\") pod \"authorino-75d556bd77-cqspn\" (UID: \"29c00da5-0951-452a-bf66-cebd7f5a156b\") " pod="kuadrant-system/authorino-75d556bd77-cqspn"
Jul 23 11:44:43.652304 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:44:43.652273    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zz7ht\" (UniqueName: \"kubernetes.io/projected/29c00da5-0951-452a-bf66-cebd7f5a156b-kube-api-access-zz7ht\") pod \"authorino-75d556bd77-cqspn\" (UID: \"29c00da5-0951-452a-bf66-cebd7f5a156b\") " pod="kuadrant-system/authorino-75d556bd77-cqspn"
Jul 23 11:44:43.662377 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:44:43.662346    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zz7ht\" (UniqueName: \"kubernetes.io/projected/29c00da5-0951-452a-bf66-cebd7f5a156b-kube-api-access-zz7ht\") pod \"authorino-75d556bd77-cqspn\" (UID: \"29c00da5-0951-452a-bf66-cebd7f5a156b\") " pod="kuadrant-system/authorino-75d556bd77-cqspn"
Jul 23 11:44:43.740375 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:44:43.740338    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-cqspn"
Jul 23 11:44:43.894035 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:44:43.894005    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-cqspn"]
Jul 23 11:44:43.897572 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:44:43.897544    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod29c00da5_0951_452a_bf66_cebd7f5a156b.slice/crio-4ec18066e791528462cd8cd9c0ee0b4f6ab2d079005be2349205cb7dce77636f WatchSource:0}: Error finding container 4ec18066e791528462cd8cd9c0ee0b4f6ab2d079005be2349205cb7dce77636f: Status 404 returned error can't find the container with id 4ec18066e791528462cd8cd9c0ee0b4f6ab2d079005be2349205cb7dce77636f
Jul 23 11:44:44.502143 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:44:44.502075    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-cqspn" event={"ID":"29c00da5-0951-452a-bf66-cebd7f5a156b","Type":"ContainerStarted","Data":"4ec18066e791528462cd8cd9c0ee0b4f6ab2d079005be2349205cb7dce77636f"}
Jul 23 11:44:48.524526 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:44:48.524483    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-97hxc" event={"ID":"cede1f68-63e6-4a1e-85d8-cbd8ac747220","Type":"ContainerStarted","Data":"82e3b084d310fcb73e3a2d789805897b69ca4a598d099b2a2296f5f1a2c170a4"}
Jul 23 11:44:48.525034 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:44:48.524547    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-97hxc"
Jul 23 11:44:48.525968 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:44:48.525941    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-cqspn" event={"ID":"29c00da5-0951-452a-bf66-cebd7f5a156b","Type":"ContainerStarted","Data":"573e9cfd4e0cce2ef65226411d0aabd996af9389abb66df60a4f4c372998ffb4"}
Jul 23 11:44:48.543007 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:44:48.542941    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-64c94fd64f-97hxc" podStartSLOduration=1.396650382 podStartE2EDuration="6.542925122s" totalImagesPullingTime="5.14627474s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 11:44:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:44:48.54163588 +0000 UTC m=+542.934573674" watchObservedRunningTime="2026-07-23 11:44:48.542925122 +0000 UTC m=+542.935862952"
Jul 23 11:44:48.556441 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:44:48.556391    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-75d556bd77-cqspn" podStartSLOduration=1.495287279 podStartE2EDuration="5.55637342s" totalImagesPullingTime="4.061086141s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 11:44:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:44:48.555698879 +0000 UTC m=+542.948636657" watchObservedRunningTime="2026-07-23 11:44:48.55637342 +0000 UTC m=+542.949311207"
Jul 23 11:44:57.900294 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:44:57.900255    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-97hxc"]
Jul 23 11:44:57.900785 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:44:57.900429    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-64c94fd64f-97hxc" podUID="cede1f68-63e6-4a1e-85d8-cbd8ac747220" containerName="limitador" containerID="cri-o://82e3b084d310fcb73e3a2d789805897b69ca4a598d099b2a2296f5f1a2c170a4" gracePeriod=30
Jul 23 11:44:57.902780 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:44:57.902755    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-97hxc"
Jul 23 11:44:58.451989 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:44:58.451961    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-97hxc"
Jul 23 11:44:58.477113 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:44:58.477031    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/cede1f68-63e6-4a1e-85d8-cbd8ac747220-config-file\") pod \"cede1f68-63e6-4a1e-85d8-cbd8ac747220\" (UID: \"cede1f68-63e6-4a1e-85d8-cbd8ac747220\") "
Jul 23 11:44:58.477113 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:44:58.477077    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9ds2s\" (UniqueName: \"kubernetes.io/projected/cede1f68-63e6-4a1e-85d8-cbd8ac747220-kube-api-access-9ds2s\") pod \"cede1f68-63e6-4a1e-85d8-cbd8ac747220\" (UID: \"cede1f68-63e6-4a1e-85d8-cbd8ac747220\") "
Jul 23 11:44:58.477441 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:44:58.477413    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cede1f68-63e6-4a1e-85d8-cbd8ac747220-config-file" (OuterVolumeSpecName: "config-file") pod "cede1f68-63e6-4a1e-85d8-cbd8ac747220" (UID: "cede1f68-63e6-4a1e-85d8-cbd8ac747220"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 11:44:58.479358 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:44:58.479334    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cede1f68-63e6-4a1e-85d8-cbd8ac747220-kube-api-access-9ds2s" (OuterVolumeSpecName: "kube-api-access-9ds2s") pod "cede1f68-63e6-4a1e-85d8-cbd8ac747220" (UID: "cede1f68-63e6-4a1e-85d8-cbd8ac747220"). InnerVolumeSpecName "kube-api-access-9ds2s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 11:44:58.568576 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:44:58.568535    2563 generic.go:356] "Generic (PLEG): container finished" podID="cede1f68-63e6-4a1e-85d8-cbd8ac747220" containerID="82e3b084d310fcb73e3a2d789805897b69ca4a598d099b2a2296f5f1a2c170a4" exitCode=0
Jul 23 11:44:58.568576 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:44:58.568578    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-97hxc"
Jul 23 11:44:58.568796 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:44:58.568596    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-97hxc" event={"ID":"cede1f68-63e6-4a1e-85d8-cbd8ac747220","Type":"ContainerDied","Data":"82e3b084d310fcb73e3a2d789805897b69ca4a598d099b2a2296f5f1a2c170a4"}
Jul 23 11:44:58.568796 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:44:58.568649    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-97hxc" event={"ID":"cede1f68-63e6-4a1e-85d8-cbd8ac747220","Type":"ContainerDied","Data":"04331138ab910c1c5243d54da7c4c58e5389088148d8ead2fbfd98cdef8c180c"}
Jul 23 11:44:58.568796 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:44:58.568674    2563 scope.go:117] "RemoveContainer" containerID="82e3b084d310fcb73e3a2d789805897b69ca4a598d099b2a2296f5f1a2c170a4"
Jul 23 11:44:58.577401 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:44:58.577382    2563 scope.go:117] "RemoveContainer" containerID="82e3b084d310fcb73e3a2d789805897b69ca4a598d099b2a2296f5f1a2c170a4"
Jul 23 11:44:58.577583 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:44:58.577563    2563 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/cede1f68-63e6-4a1e-85d8-cbd8ac747220-config-file\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 23 11:44:58.577643 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:44:58.577589    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9ds2s\" (UniqueName: \"kubernetes.io/projected/cede1f68-63e6-4a1e-85d8-cbd8ac747220-kube-api-access-9ds2s\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 23 11:44:58.577740 ip-10-0-141-229 kubenswrapper[2563]: E0723 11:44:58.577654    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82e3b084d310fcb73e3a2d789805897b69ca4a598d099b2a2296f5f1a2c170a4\": container with ID starting with 82e3b084d310fcb73e3a2d789805897b69ca4a598d099b2a2296f5f1a2c170a4 not found: ID does not exist" containerID="82e3b084d310fcb73e3a2d789805897b69ca4a598d099b2a2296f5f1a2c170a4"
Jul 23 11:44:58.577740 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:44:58.577706    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82e3b084d310fcb73e3a2d789805897b69ca4a598d099b2a2296f5f1a2c170a4"} err="failed to get container status \"82e3b084d310fcb73e3a2d789805897b69ca4a598d099b2a2296f5f1a2c170a4\": rpc error: code = NotFound desc = could not find container \"82e3b084d310fcb73e3a2d789805897b69ca4a598d099b2a2296f5f1a2c170a4\": container with ID starting with 82e3b084d310fcb73e3a2d789805897b69ca4a598d099b2a2296f5f1a2c170a4 not found: ID does not exist"
Jul 23 11:44:58.589650 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:44:58.589620    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-97hxc"]
Jul 23 11:44:58.591610 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:44:58.591593    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-97hxc"]
Jul 23 11:45:00.077827 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:45:00.077792    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cede1f68-63e6-4a1e-85d8-cbd8ac747220" path="/var/lib/kubelet/pods/cede1f68-63e6-4a1e-85d8-cbd8ac747220/volumes"
Jul 23 11:45:02.467205 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:45:02.467167    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:45:03.027451 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:45:03.027416    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:45:03.256914 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:45:03.256882    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:45:15.597399 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:45:15.597355    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:45:15.852355 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:45:15.852269    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:45:17.851384 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:45:17.851349    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:45:18.285853 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:45:18.285819    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-cqspn"]
Jul 23 11:45:18.286029 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:45:18.286003    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-75d556bd77-cqspn" podUID="29c00da5-0951-452a-bf66-cebd7f5a156b" containerName="authorino" containerID="cri-o://573e9cfd4e0cce2ef65226411d0aabd996af9389abb66df60a4f4c372998ffb4" gracePeriod=30
Jul 23 11:45:18.534498 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:45:18.534473    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-cqspn"
Jul 23 11:45:18.652051 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:45:18.652020    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zz7ht\" (UniqueName: \"kubernetes.io/projected/29c00da5-0951-452a-bf66-cebd7f5a156b-kube-api-access-zz7ht\") pod \"29c00da5-0951-452a-bf66-cebd7f5a156b\" (UID: \"29c00da5-0951-452a-bf66-cebd7f5a156b\") "
Jul 23 11:45:18.654271 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:45:18.654240    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29c00da5-0951-452a-bf66-cebd7f5a156b-kube-api-access-zz7ht" (OuterVolumeSpecName: "kube-api-access-zz7ht") pod "29c00da5-0951-452a-bf66-cebd7f5a156b" (UID: "29c00da5-0951-452a-bf66-cebd7f5a156b"). InnerVolumeSpecName "kube-api-access-zz7ht". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 11:45:18.657039 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:45:18.657011    2563 generic.go:356] "Generic (PLEG): container finished" podID="29c00da5-0951-452a-bf66-cebd7f5a156b" containerID="573e9cfd4e0cce2ef65226411d0aabd996af9389abb66df60a4f4c372998ffb4" exitCode=0
Jul 23 11:45:18.657145 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:45:18.657055    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-cqspn"
Jul 23 11:45:18.657145 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:45:18.657097    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-cqspn" event={"ID":"29c00da5-0951-452a-bf66-cebd7f5a156b","Type":"ContainerDied","Data":"573e9cfd4e0cce2ef65226411d0aabd996af9389abb66df60a4f4c372998ffb4"}
Jul 23 11:45:18.657145 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:45:18.657138    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-cqspn" event={"ID":"29c00da5-0951-452a-bf66-cebd7f5a156b","Type":"ContainerDied","Data":"4ec18066e791528462cd8cd9c0ee0b4f6ab2d079005be2349205cb7dce77636f"}
Jul 23 11:45:18.657242 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:45:18.657156    2563 scope.go:117] "RemoveContainer" containerID="573e9cfd4e0cce2ef65226411d0aabd996af9389abb66df60a4f4c372998ffb4"
Jul 23 11:45:18.665790 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:45:18.665761    2563 scope.go:117] "RemoveContainer" containerID="573e9cfd4e0cce2ef65226411d0aabd996af9389abb66df60a4f4c372998ffb4"
Jul 23 11:45:18.666053 ip-10-0-141-229 kubenswrapper[2563]: E0723 11:45:18.666037    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"573e9cfd4e0cce2ef65226411d0aabd996af9389abb66df60a4f4c372998ffb4\": container with ID starting with 573e9cfd4e0cce2ef65226411d0aabd996af9389abb66df60a4f4c372998ffb4 not found: ID does not exist" containerID="573e9cfd4e0cce2ef65226411d0aabd996af9389abb66df60a4f4c372998ffb4"
Jul 23 11:45:18.666094 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:45:18.666061    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"573e9cfd4e0cce2ef65226411d0aabd996af9389abb66df60a4f4c372998ffb4"} err="failed to get container status \"573e9cfd4e0cce2ef65226411d0aabd996af9389abb66df60a4f4c372998ffb4\": rpc error: code = NotFound desc = could not find container \"573e9cfd4e0cce2ef65226411d0aabd996af9389abb66df60a4f4c372998ffb4\": container with ID starting with 573e9cfd4e0cce2ef65226411d0aabd996af9389abb66df60a4f4c372998ffb4 not found: ID does not exist"
Jul 23 11:45:18.677223 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:45:18.677191    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-cqspn"]
Jul 23 11:45:18.679203 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:45:18.679180    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-75d556bd77-cqspn"]
Jul 23 11:45:18.753179 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:45:18.753145    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zz7ht\" (UniqueName: \"kubernetes.io/projected/29c00da5-0951-452a-bf66-cebd7f5a156b-kube-api-access-zz7ht\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 23 11:45:20.076845 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:45:20.076810    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="29c00da5-0951-452a-bf66-cebd7f5a156b" path="/var/lib/kubelet/pods/29c00da5-0951-452a-bf66-cebd7f5a156b/volumes"
Jul 23 11:45:23.178495 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:45:23.178462    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:45:23.202891 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:45:23.202856    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6d6f87dcc4-k5cjl"]
Jul 23 11:45:23.203202 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:45:23.203188    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="29c00da5-0951-452a-bf66-cebd7f5a156b" containerName="authorino"
Jul 23 11:45:23.203250 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:45:23.203207    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="29c00da5-0951-452a-bf66-cebd7f5a156b" containerName="authorino"
Jul 23 11:45:23.203250 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:45:23.203219    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cede1f68-63e6-4a1e-85d8-cbd8ac747220" containerName="limitador"
Jul 23 11:45:23.203250 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:45:23.203227    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="cede1f68-63e6-4a1e-85d8-cbd8ac747220" containerName="limitador"
Jul 23 11:45:23.203343 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:45:23.203285    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="29c00da5-0951-452a-bf66-cebd7f5a156b" containerName="authorino"
Jul 23 11:45:23.203343 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:45:23.203299    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="cede1f68-63e6-4a1e-85d8-cbd8ac747220" containerName="limitador"
Jul 23 11:45:23.210248 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:45:23.210227    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6d6f87dcc4-k5cjl"
Jul 23 11:45:23.214712 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:45:23.214664    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Jul 23 11:45:23.214903 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:45:23.214885    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-nwgls\""
Jul 23 11:45:23.215903 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:45:23.215880    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Jul 23 11:45:23.222854 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:45:23.222827    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6d6f87dcc4-k5cjl"]
Jul 23 11:45:23.291965 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:45:23.291924    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/968dec29-e656-4a48-97f4-7927b78b0ec7-webhook-cert\") pod \"maas-controller-6d6f87dcc4-k5cjl\" (UID: \"968dec29-e656-4a48-97f4-7927b78b0ec7\") " pod="opendatahub/maas-controller-6d6f87dcc4-k5cjl"
Jul 23 11:45:23.292153 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:45:23.291999    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hvf7\" (UniqueName: \"kubernetes.io/projected/968dec29-e656-4a48-97f4-7927b78b0ec7-kube-api-access-8hvf7\") pod \"maas-controller-6d6f87dcc4-k5cjl\" (UID: \"968dec29-e656-4a48-97f4-7927b78b0ec7\") " pod="opendatahub/maas-controller-6d6f87dcc4-k5cjl"
Jul 23 11:45:23.383347 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:45:23.383296    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:45:23.392440 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:45:23.392414    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/968dec29-e656-4a48-97f4-7927b78b0ec7-webhook-cert\") pod \"maas-controller-6d6f87dcc4-k5cjl\" (UID: \"968dec29-e656-4a48-97f4-7927b78b0ec7\") " pod="opendatahub/maas-controller-6d6f87dcc4-k5cjl"
Jul 23 11:45:23.392553 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:45:23.392497    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hvf7\" (UniqueName: \"kubernetes.io/projected/968dec29-e656-4a48-97f4-7927b78b0ec7-kube-api-access-8hvf7\") pod \"maas-controller-6d6f87dcc4-k5cjl\" (UID: \"968dec29-e656-4a48-97f4-7927b78b0ec7\") " pod="opendatahub/maas-controller-6d6f87dcc4-k5cjl"
Jul 23 11:45:23.395409 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:45:23.395375    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/968dec29-e656-4a48-97f4-7927b78b0ec7-webhook-cert\") pod \"maas-controller-6d6f87dcc4-k5cjl\" (UID: \"968dec29-e656-4a48-97f4-7927b78b0ec7\") " pod="opendatahub/maas-controller-6d6f87dcc4-k5cjl"
Jul 23 11:45:23.419347 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:45:23.419317    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hvf7\" (UniqueName: \"kubernetes.io/projected/968dec29-e656-4a48-97f4-7927b78b0ec7-kube-api-access-8hvf7\") pod \"maas-controller-6d6f87dcc4-k5cjl\" (UID: \"968dec29-e656-4a48-97f4-7927b78b0ec7\") " pod="opendatahub/maas-controller-6d6f87dcc4-k5cjl"
Jul 23 11:45:23.481256 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:45:23.481177    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-78746d79b5-w9j78"]
Jul 23 11:45:23.483617 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:45:23.483601    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-78746d79b5-w9j78"
Jul 23 11:45:23.504565 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:45:23.504540    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-78746d79b5-w9j78"]
Jul 23 11:45:23.519854 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:45:23.519827    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6d6f87dcc4-k5cjl"
Jul 23 11:45:23.594131 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:45:23.594096    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c0c873c3-6ff4-4f3f-a81f-0d3b1fb87307-webhook-cert\") pod \"maas-controller-78746d79b5-w9j78\" (UID: \"c0c873c3-6ff4-4f3f-a81f-0d3b1fb87307\") " pod="opendatahub/maas-controller-78746d79b5-w9j78"
Jul 23 11:45:23.594308 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:45:23.594182    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4qzw\" (UniqueName: \"kubernetes.io/projected/c0c873c3-6ff4-4f3f-a81f-0d3b1fb87307-kube-api-access-z4qzw\") pod \"maas-controller-78746d79b5-w9j78\" (UID: \"c0c873c3-6ff4-4f3f-a81f-0d3b1fb87307\") " pod="opendatahub/maas-controller-78746d79b5-w9j78"
Jul 23 11:45:23.672392 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:45:23.672358    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6d6f87dcc4-k5cjl"]
Jul 23 11:45:23.675917 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:45:23.675886    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod968dec29_e656_4a48_97f4_7927b78b0ec7.slice/crio-4ba7753e4b574b78d5595561789b5be9e89f34ee9cd7e30f13da627333b25fb0 WatchSource:0}: Error finding container 4ba7753e4b574b78d5595561789b5be9e89f34ee9cd7e30f13da627333b25fb0: Status 404 returned error can't find the container with id 4ba7753e4b574b78d5595561789b5be9e89f34ee9cd7e30f13da627333b25fb0
Jul 23 11:45:23.694728 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:45:23.694701    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c0c873c3-6ff4-4f3f-a81f-0d3b1fb87307-webhook-cert\") pod \"maas-controller-78746d79b5-w9j78\" (UID: \"c0c873c3-6ff4-4f3f-a81f-0d3b1fb87307\") " pod="opendatahub/maas-controller-78746d79b5-w9j78"
Jul 23 11:45:23.694842 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:45:23.694795    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4qzw\" (UniqueName: \"kubernetes.io/projected/c0c873c3-6ff4-4f3f-a81f-0d3b1fb87307-kube-api-access-z4qzw\") pod \"maas-controller-78746d79b5-w9j78\" (UID: \"c0c873c3-6ff4-4f3f-a81f-0d3b1fb87307\") " pod="opendatahub/maas-controller-78746d79b5-w9j78"
Jul 23 11:45:23.697248 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:45:23.697227    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c0c873c3-6ff4-4f3f-a81f-0d3b1fb87307-webhook-cert\") pod \"maas-controller-78746d79b5-w9j78\" (UID: \"c0c873c3-6ff4-4f3f-a81f-0d3b1fb87307\") " pod="opendatahub/maas-controller-78746d79b5-w9j78"
Jul 23 11:45:23.711502 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:45:23.711475    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4qzw\" (UniqueName: \"kubernetes.io/projected/c0c873c3-6ff4-4f3f-a81f-0d3b1fb87307-kube-api-access-z4qzw\") pod \"maas-controller-78746d79b5-w9j78\" (UID: \"c0c873c3-6ff4-4f3f-a81f-0d3b1fb87307\") " pod="opendatahub/maas-controller-78746d79b5-w9j78"
Jul 23 11:45:23.750860 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:45:23.750771    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:45:23.793620 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:45:23.793571    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-78746d79b5-w9j78"
Jul 23 11:45:23.929911 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:45:23.929882    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-78746d79b5-w9j78"]
Jul 23 11:45:23.932653 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:45:23.932625    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0c873c3_6ff4_4f3f_a81f_0d3b1fb87307.slice/crio-e59a9c49209d4903ad35fe0da1f4cc011c40dbbf7859d59d71715632fa33d31b WatchSource:0}: Error finding container e59a9c49209d4903ad35fe0da1f4cc011c40dbbf7859d59d71715632fa33d31b: Status 404 returned error can't find the container with id e59a9c49209d4903ad35fe0da1f4cc011c40dbbf7859d59d71715632fa33d31b
Jul 23 11:45:24.692890 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:45:24.692845    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-78746d79b5-w9j78" event={"ID":"c0c873c3-6ff4-4f3f-a81f-0d3b1fb87307","Type":"ContainerStarted","Data":"e59a9c49209d4903ad35fe0da1f4cc011c40dbbf7859d59d71715632fa33d31b"}
Jul 23 11:45:24.698285 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:45:24.698250    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6d6f87dcc4-k5cjl" event={"ID":"968dec29-e656-4a48-97f4-7927b78b0ec7","Type":"ContainerStarted","Data":"4ba7753e4b574b78d5595561789b5be9e89f34ee9cd7e30f13da627333b25fb0"}
Jul 23 11:45:26.489383 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:45:26.489348    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:45:26.709476 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:45:26.709442    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-78746d79b5-w9j78" event={"ID":"c0c873c3-6ff4-4f3f-a81f-0d3b1fb87307","Type":"ContainerStarted","Data":"02245d0c8e15d36befbb783cede1a59535212ee27fbf441e49b9e7fded7244de"}
Jul 23 11:45:26.709703 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:45:26.709613    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-78746d79b5-w9j78"
Jul 23 11:45:26.710893 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:45:26.710872    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6d6f87dcc4-k5cjl" event={"ID":"968dec29-e656-4a48-97f4-7927b78b0ec7","Type":"ContainerStarted","Data":"5483d39e56b86801e106007e0db63b5314a49676a70618db53d3af5b2987b4b0"}
Jul 23 11:45:26.711018 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:45:26.711004    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6d6f87dcc4-k5cjl"
Jul 23 11:45:26.766267 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:45:26.766234    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:45:26.907545 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:45:26.907491    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-6d6f87dcc4-k5cjl" podStartSLOduration=1.2779717609999999 podStartE2EDuration="3.907475901s" totalImagesPullingTime="2.62950414s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 11:45:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:45:26.906378195 +0000 UTC m=+581.299315983" watchObservedRunningTime="2026-07-23 11:45:26.907475901 +0000 UTC m=+581.300413683"
Jul 23 11:45:26.907753 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:45:26.907582    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-78746d79b5-w9j78" podStartSLOduration=1.5255673669999998 podStartE2EDuration="3.907577325s" totalImagesPullingTime="2.382009958s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 11:45:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:45:26.785800921 +0000 UTC m=+581.178738709" watchObservedRunningTime="2026-07-23 11:45:26.907577325 +0000 UTC m=+581.300515114"
Jul 23 11:45:37.718362 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:45:37.718332    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6d6f87dcc4-k5cjl"
Jul 23 11:45:37.718779 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:45:37.718404    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-78746d79b5-w9j78"
Jul 23 11:45:37.784343 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:45:37.784310    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6d6f87dcc4-k5cjl"]
Jul 23 11:45:37.784527 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:45:37.784497    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-6d6f87dcc4-k5cjl" podUID="968dec29-e656-4a48-97f4-7927b78b0ec7" containerName="manager" containerID="cri-o://5483d39e56b86801e106007e0db63b5314a49676a70618db53d3af5b2987b4b0" gracePeriod=30
Jul 23 11:45:38.024948 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:45:38.024905    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6d6f87dcc4-k5cjl"
Jul 23 11:45:38.129489 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:45:38.129451    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/968dec29-e656-4a48-97f4-7927b78b0ec7-webhook-cert\") pod \"968dec29-e656-4a48-97f4-7927b78b0ec7\" (UID: \"968dec29-e656-4a48-97f4-7927b78b0ec7\") "
Jul 23 11:45:38.129665 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:45:38.129509    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8hvf7\" (UniqueName: \"kubernetes.io/projected/968dec29-e656-4a48-97f4-7927b78b0ec7-kube-api-access-8hvf7\") pod \"968dec29-e656-4a48-97f4-7927b78b0ec7\" (UID: \"968dec29-e656-4a48-97f4-7927b78b0ec7\") "
Jul 23 11:45:38.131841 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:45:38.131812    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/968dec29-e656-4a48-97f4-7927b78b0ec7-kube-api-access-8hvf7" (OuterVolumeSpecName: "kube-api-access-8hvf7") pod "968dec29-e656-4a48-97f4-7927b78b0ec7" (UID: "968dec29-e656-4a48-97f4-7927b78b0ec7"). InnerVolumeSpecName "kube-api-access-8hvf7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 11:45:38.131952 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:45:38.131849    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/968dec29-e656-4a48-97f4-7927b78b0ec7-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "968dec29-e656-4a48-97f4-7927b78b0ec7" (UID: "968dec29-e656-4a48-97f4-7927b78b0ec7"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 11:45:38.230566 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:45:38.230524    2563 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/968dec29-e656-4a48-97f4-7927b78b0ec7-webhook-cert\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 23 11:45:38.230782 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:45:38.230575    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8hvf7\" (UniqueName: \"kubernetes.io/projected/968dec29-e656-4a48-97f4-7927b78b0ec7-kube-api-access-8hvf7\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 23 11:45:38.765875 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:45:38.765836    2563 generic.go:356] "Generic (PLEG): container finished" podID="968dec29-e656-4a48-97f4-7927b78b0ec7" containerID="5483d39e56b86801e106007e0db63b5314a49676a70618db53d3af5b2987b4b0" exitCode=0
Jul 23 11:45:38.766933 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:45:38.765888    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6d6f87dcc4-k5cjl"
Jul 23 11:45:38.766933 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:45:38.765896    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6d6f87dcc4-k5cjl" event={"ID":"968dec29-e656-4a48-97f4-7927b78b0ec7","Type":"ContainerDied","Data":"5483d39e56b86801e106007e0db63b5314a49676a70618db53d3af5b2987b4b0"}
Jul 23 11:45:38.766933 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:45:38.765938    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6d6f87dcc4-k5cjl" event={"ID":"968dec29-e656-4a48-97f4-7927b78b0ec7","Type":"ContainerDied","Data":"4ba7753e4b574b78d5595561789b5be9e89f34ee9cd7e30f13da627333b25fb0"}
Jul 23 11:45:38.766933 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:45:38.765959    2563 scope.go:117] "RemoveContainer" containerID="5483d39e56b86801e106007e0db63b5314a49676a70618db53d3af5b2987b4b0"
Jul 23 11:45:38.774660 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:45:38.774644    2563 scope.go:117] "RemoveContainer" containerID="5483d39e56b86801e106007e0db63b5314a49676a70618db53d3af5b2987b4b0"
Jul 23 11:45:38.774937 ip-10-0-141-229 kubenswrapper[2563]: E0723 11:45:38.774922    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5483d39e56b86801e106007e0db63b5314a49676a70618db53d3af5b2987b4b0\": container with ID starting with 5483d39e56b86801e106007e0db63b5314a49676a70618db53d3af5b2987b4b0 not found: ID does not exist" containerID="5483d39e56b86801e106007e0db63b5314a49676a70618db53d3af5b2987b4b0"
Jul 23 11:45:38.774983 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:45:38.774946    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5483d39e56b86801e106007e0db63b5314a49676a70618db53d3af5b2987b4b0"} err="failed to get container status \"5483d39e56b86801e106007e0db63b5314a49676a70618db53d3af5b2987b4b0\": rpc error: code = NotFound desc = could not find container \"5483d39e56b86801e106007e0db63b5314a49676a70618db53d3af5b2987b4b0\": container with ID starting with 5483d39e56b86801e106007e0db63b5314a49676a70618db53d3af5b2987b4b0 not found: ID does not exist"
Jul 23 11:45:38.787761 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:45:38.787731    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6d6f87dcc4-k5cjl"]
Jul 23 11:45:38.791745 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:45:38.791722    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-6d6f87dcc4-k5cjl"]
Jul 23 11:45:40.077662 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:45:40.077625    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="968dec29-e656-4a48-97f4-7927b78b0ec7" path="/var/lib/kubelet/pods/968dec29-e656-4a48-97f4-7927b78b0ec7/volumes"
Jul 23 11:45:45.985015 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:45:45.984986    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nhm9p_93649ac9-9062-46c0-adaa-1048d471211a/ovn-acl-logging/0.log"
Jul 23 11:45:45.988018 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:45:45.987985    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nhm9p_93649ac9-9062-46c0-adaa-1048d471211a/ovn-acl-logging/0.log"
Jul 23 11:45:48.020960 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:45:48.020914    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:45:56.095546 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:45:56.095479    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:45:56.895155 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:45:56.895119    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-64c4fb9c4-25vlh"]
Jul 23 11:45:56.895452 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:45:56.895441    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="968dec29-e656-4a48-97f4-7927b78b0ec7" containerName="manager"
Jul 23 11:45:56.895497 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:45:56.895454    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="968dec29-e656-4a48-97f4-7927b78b0ec7" containerName="manager"
Jul 23 11:45:56.895535 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:45:56.895508    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="968dec29-e656-4a48-97f4-7927b78b0ec7" containerName="manager"
Jul 23 11:45:56.898502 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:45:56.898481    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-64c4fb9c4-25vlh"
Jul 23 11:45:56.902146 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:45:56.902120    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 23 11:45:56.902874 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:45:56.902856    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 23 11:45:56.902938 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:45:56.902907    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Jul 23 11:45:56.902938 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:45:56.902923    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-c9lpl\""
Jul 23 11:45:56.915264 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:45:56.915227    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-64c4fb9c4-25vlh"]
Jul 23 11:45:56.973259 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:45:56.973222    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/01034321-2fa1-4709-bbf0-23d21b05fb49-openshift-service-ca\") pod \"maas-api-64c4fb9c4-25vlh\" (UID: \"01034321-2fa1-4709-bbf0-23d21b05fb49\") " pod="odh-ai-gateway-infra/maas-api-64c4fb9c4-25vlh"
Jul 23 11:45:56.973455 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:45:56.973282    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/01034321-2fa1-4709-bbf0-23d21b05fb49-maas-api-tls\") pod \"maas-api-64c4fb9c4-25vlh\" (UID: \"01034321-2fa1-4709-bbf0-23d21b05fb49\") " pod="odh-ai-gateway-infra/maas-api-64c4fb9c4-25vlh"
Jul 23 11:45:56.973455 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:45:56.973377    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qbxpq\" (UniqueName: \"kubernetes.io/projected/01034321-2fa1-4709-bbf0-23d21b05fb49-kube-api-access-qbxpq\") pod \"maas-api-64c4fb9c4-25vlh\" (UID: \"01034321-2fa1-4709-bbf0-23d21b05fb49\") " pod="odh-ai-gateway-infra/maas-api-64c4fb9c4-25vlh"
Jul 23 11:45:57.074014 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:45:57.073972    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/01034321-2fa1-4709-bbf0-23d21b05fb49-openshift-service-ca\") pod \"maas-api-64c4fb9c4-25vlh\" (UID: \"01034321-2fa1-4709-bbf0-23d21b05fb49\") " pod="odh-ai-gateway-infra/maas-api-64c4fb9c4-25vlh"
Jul 23 11:45:57.074215 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:45:57.074030    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/01034321-2fa1-4709-bbf0-23d21b05fb49-maas-api-tls\") pod \"maas-api-64c4fb9c4-25vlh\" (UID: \"01034321-2fa1-4709-bbf0-23d21b05fb49\") " pod="odh-ai-gateway-infra/maas-api-64c4fb9c4-25vlh"
Jul 23 11:45:57.074215 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:45:57.074107    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbxpq\" (UniqueName: \"kubernetes.io/projected/01034321-2fa1-4709-bbf0-23d21b05fb49-kube-api-access-qbxpq\") pod \"maas-api-64c4fb9c4-25vlh\" (UID: \"01034321-2fa1-4709-bbf0-23d21b05fb49\") " pod="odh-ai-gateway-infra/maas-api-64c4fb9c4-25vlh"
Jul 23 11:45:57.074769 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:45:57.074742    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/01034321-2fa1-4709-bbf0-23d21b05fb49-openshift-service-ca\") pod \"maas-api-64c4fb9c4-25vlh\" (UID: \"01034321-2fa1-4709-bbf0-23d21b05fb49\") " pod="odh-ai-gateway-infra/maas-api-64c4fb9c4-25vlh"
Jul 23 11:45:57.076647 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:45:57.076623    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/01034321-2fa1-4709-bbf0-23d21b05fb49-maas-api-tls\") pod \"maas-api-64c4fb9c4-25vlh\" (UID: \"01034321-2fa1-4709-bbf0-23d21b05fb49\") " pod="odh-ai-gateway-infra/maas-api-64c4fb9c4-25vlh"
Jul 23 11:45:57.087748 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:45:57.087707    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbxpq\" (UniqueName: \"kubernetes.io/projected/01034321-2fa1-4709-bbf0-23d21b05fb49-kube-api-access-qbxpq\") pod \"maas-api-64c4fb9c4-25vlh\" (UID: \"01034321-2fa1-4709-bbf0-23d21b05fb49\") " pod="odh-ai-gateway-infra/maas-api-64c4fb9c4-25vlh"
Jul 23 11:45:57.209445 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:45:57.209331    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-64c4fb9c4-25vlh"
Jul 23 11:45:57.366278 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:45:57.366209    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-64c4fb9c4-25vlh"]
Jul 23 11:45:57.368940 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:45:57.368908    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01034321_2fa1_4709_bbf0_23d21b05fb49.slice/crio-a8783fc72e7e09c177d95ade38c5465d872b75550414225fb37cdfd9eda6f66d WatchSource:0}: Error finding container a8783fc72e7e09c177d95ade38c5465d872b75550414225fb37cdfd9eda6f66d: Status 404 returned error can't find the container with id a8783fc72e7e09c177d95ade38c5465d872b75550414225fb37cdfd9eda6f66d
Jul 23 11:45:57.370947 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:45:57.370919    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 23 11:45:57.393444 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:45:57.393404    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7677b774bb-8mlt4"]
Jul 23 11:45:57.398057 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:45:57.398024    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7677b774bb-8mlt4"
Jul 23 11:45:57.400523 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:45:57.400493    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-9dc54\""
Jul 23 11:45:57.400724 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:45:57.400493    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Jul 23 11:45:57.409450 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:45:57.409417    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7677b774bb-8mlt4"]
Jul 23 11:45:57.415058 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:45:57.415023    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5bf47cd988-qz6rk"]
Jul 23 11:45:57.418621 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:45:57.418591    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5bf47cd988-qz6rk"
Jul 23 11:45:57.428280 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:45:57.428248    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5bf47cd988-qz6rk"]
Jul 23 11:45:57.478715 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:45:57.478582    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bfa0456d-4cdc-431d-b86b-075f929d20de-plugins-config-volume\") pod \"payload-pre-processing-7677b774bb-8mlt4\" (UID: \"bfa0456d-4cdc-431d-b86b-075f929d20de\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-8mlt4"
Jul 23 11:45:57.478715 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:45:57.478655    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqrkl\" (UniqueName: \"kubernetes.io/projected/0cca871f-490f-49cc-8782-a47403080a0d-kube-api-access-wqrkl\") pod \"payload-processing-5bf47cd988-qz6rk\" (UID: \"0cca871f-490f-49cc-8782-a47403080a0d\") " pod="openshift-ingress/payload-processing-5bf47cd988-qz6rk"
Jul 23 11:45:57.478924 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:45:57.478740    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxwtv\" (UniqueName: \"kubernetes.io/projected/bfa0456d-4cdc-431d-b86b-075f929d20de-kube-api-access-gxwtv\") pod \"payload-pre-processing-7677b774bb-8mlt4\" (UID: \"bfa0456d-4cdc-431d-b86b-075f929d20de\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-8mlt4"
Jul 23 11:45:57.478924 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:45:57.478776    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0cca871f-490f-49cc-8782-a47403080a0d-plugins-config-volume\") pod \"payload-processing-5bf47cd988-qz6rk\" (UID: \"0cca871f-490f-49cc-8782-a47403080a0d\") " pod="openshift-ingress/payload-processing-5bf47cd988-qz6rk"
Jul 23 11:45:57.579346 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:45:57.579312    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqrkl\" (UniqueName: \"kubernetes.io/projected/0cca871f-490f-49cc-8782-a47403080a0d-kube-api-access-wqrkl\") pod \"payload-processing-5bf47cd988-qz6rk\" (UID: \"0cca871f-490f-49cc-8782-a47403080a0d\") " pod="openshift-ingress/payload-processing-5bf47cd988-qz6rk"
Jul 23 11:45:57.579346 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:45:57.579351    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxwtv\" (UniqueName: \"kubernetes.io/projected/bfa0456d-4cdc-431d-b86b-075f929d20de-kube-api-access-gxwtv\") pod \"payload-pre-processing-7677b774bb-8mlt4\" (UID: \"bfa0456d-4cdc-431d-b86b-075f929d20de\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-8mlt4"
Jul 23 11:45:57.579628 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:45:57.579371    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0cca871f-490f-49cc-8782-a47403080a0d-plugins-config-volume\") pod \"payload-processing-5bf47cd988-qz6rk\" (UID: \"0cca871f-490f-49cc-8782-a47403080a0d\") " pod="openshift-ingress/payload-processing-5bf47cd988-qz6rk"
Jul 23 11:45:57.579628 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:45:57.579425    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bfa0456d-4cdc-431d-b86b-075f929d20de-plugins-config-volume\") pod \"payload-pre-processing-7677b774bb-8mlt4\" (UID: \"bfa0456d-4cdc-431d-b86b-075f929d20de\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-8mlt4"
Jul 23 11:45:57.580427 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:45:57.580400    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bfa0456d-4cdc-431d-b86b-075f929d20de-plugins-config-volume\") pod \"payload-pre-processing-7677b774bb-8mlt4\" (UID: \"bfa0456d-4cdc-431d-b86b-075f929d20de\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-8mlt4"
Jul 23 11:45:57.580491 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:45:57.580400    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0cca871f-490f-49cc-8782-a47403080a0d-plugins-config-volume\") pod \"payload-processing-5bf47cd988-qz6rk\" (UID: \"0cca871f-490f-49cc-8782-a47403080a0d\") " pod="openshift-ingress/payload-processing-5bf47cd988-qz6rk"
Jul 23 11:45:57.596517 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:45:57.596485    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqrkl\" (UniqueName: \"kubernetes.io/projected/0cca871f-490f-49cc-8782-a47403080a0d-kube-api-access-wqrkl\") pod \"payload-processing-5bf47cd988-qz6rk\" (UID: \"0cca871f-490f-49cc-8782-a47403080a0d\") " pod="openshift-ingress/payload-processing-5bf47cd988-qz6rk"
Jul 23 11:45:57.600091 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:45:57.600059    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxwtv\" (UniqueName: \"kubernetes.io/projected/bfa0456d-4cdc-431d-b86b-075f929d20de-kube-api-access-gxwtv\") pod \"payload-pre-processing-7677b774bb-8mlt4\" (UID: \"bfa0456d-4cdc-431d-b86b-075f929d20de\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-8mlt4"
Jul 23 11:45:57.710834 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:45:57.710790    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7677b774bb-8mlt4"
Jul 23 11:45:57.732268 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:45:57.732175    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5bf47cd988-qz6rk"
Jul 23 11:45:57.853906 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:45:57.853866    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-64c4fb9c4-25vlh" event={"ID":"01034321-2fa1-4709-bbf0-23d21b05fb49","Type":"ContainerStarted","Data":"a8783fc72e7e09c177d95ade38c5465d872b75550414225fb37cdfd9eda6f66d"}
Jul 23 11:45:57.870517 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:45:57.870482    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7677b774bb-8mlt4"]
Jul 23 11:45:57.872195 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:45:57.872163    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfa0456d_4cdc_431d_b86b_075f929d20de.slice/crio-6d3a947513677d81f6b95b8dcffb35dbc78b882fe6a69c75d7c3bb6ca01891de WatchSource:0}: Error finding container 6d3a947513677d81f6b95b8dcffb35dbc78b882fe6a69c75d7c3bb6ca01891de: Status 404 returned error can't find the container with id 6d3a947513677d81f6b95b8dcffb35dbc78b882fe6a69c75d7c3bb6ca01891de
Jul 23 11:45:57.893419 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:45:57.893242    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5bf47cd988-qz6rk"]
Jul 23 11:45:57.896220 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:45:57.896191    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0cca871f_490f_49cc_8782_a47403080a0d.slice/crio-192043769cdb649fd9b9dc1edc1425c68d85a03fda6c8d84d20db199716abcb6 WatchSource:0}: Error finding container 192043769cdb649fd9b9dc1edc1425c68d85a03fda6c8d84d20db199716abcb6: Status 404 returned error can't find the container with id 192043769cdb649fd9b9dc1edc1425c68d85a03fda6c8d84d20db199716abcb6
Jul 23 11:45:58.055604 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:45:58.055459    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:45:58.865621 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:45:58.865572    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5bf47cd988-qz6rk" event={"ID":"0cca871f-490f-49cc-8782-a47403080a0d","Type":"ContainerStarted","Data":"192043769cdb649fd9b9dc1edc1425c68d85a03fda6c8d84d20db199716abcb6"}
Jul 23 11:45:58.868312 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:45:58.868266    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7677b774bb-8mlt4" event={"ID":"bfa0456d-4cdc-431d-b86b-075f929d20de","Type":"ContainerStarted","Data":"6d3a947513677d81f6b95b8dcffb35dbc78b882fe6a69c75d7c3bb6ca01891de"}
Jul 23 11:45:59.876586 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:45:59.876542    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-64c4fb9c4-25vlh" event={"ID":"01034321-2fa1-4709-bbf0-23d21b05fb49","Type":"ContainerStarted","Data":"534ff1d23ed61faeb998bc821e1c64b94a3b5a486aa40da7cfd20aa2a096d233"}
Jul 23 11:45:59.877086 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:45:59.876774    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-64c4fb9c4-25vlh"
Jul 23 11:45:59.878706 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:45:59.878657    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7677b774bb-8mlt4" event={"ID":"bfa0456d-4cdc-431d-b86b-075f929d20de","Type":"ContainerStarted","Data":"6d06853d0bc326c47b5244d3035f9b94f00ed24d6dcde5a07c68e2b29f6fbd39"}
Jul 23 11:45:59.878921 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:45:59.878882    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7677b774bb-8mlt4"
Jul 23 11:45:59.894752 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:45:59.894674    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-64c4fb9c4-25vlh" podStartSLOduration=2.14948137 podStartE2EDuration="3.894656802s" totalImagesPullingTime="1.745175432s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 11:45:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:45:59.893440226 +0000 UTC m=+614.286378028" watchObservedRunningTime="2026-07-23 11:45:59.894656802 +0000 UTC m=+614.287594590"
Jul 23 11:45:59.910807 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:45:59.910744    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7677b774bb-8mlt4" podStartSLOduration=1.106709467 podStartE2EDuration="2.910723528s" totalImagesPullingTime="1.804014061s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 11:45:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:45:59.910400561 +0000 UTC m=+614.303338350" watchObservedRunningTime="2026-07-23 11:45:59.910723528 +0000 UTC m=+614.303661316"
Jul 23 11:46:00.885294 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:46:00.885254    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5bf47cd988-qz6rk" event={"ID":"0cca871f-490f-49cc-8782-a47403080a0d","Type":"ContainerStarted","Data":"161e581001ca2cfba1d24e47849f494c55eb1b5921798f85c8f525edaecb9f2f"}
Jul 23 11:46:00.903217 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:46:00.903148    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5bf47cd988-qz6rk" podStartSLOduration=1.8782999409999999 podStartE2EDuration="3.90312771s" totalImagesPullingTime="2.024827769s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 11:45:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:46:00.901828551 +0000 UTC m=+615.294766342" watchObservedRunningTime="2026-07-23 11:46:00.90312771 +0000 UTC m=+615.296065517"
Jul 23 11:46:01.889907 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:46:01.889868    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5bf47cd988-qz6rk"
Jul 23 11:46:05.892333 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:46:05.892294    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-64c4fb9c4-25vlh"
Jul 23 11:46:06.959540 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:46:06.959440    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:46:07.056383 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:46:07.056340    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:46:10.887798 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:46:10.887761    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7677b774bb-8mlt4"
Jul 23 11:46:12.897253 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:46:12.897213    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5bf47cd988-qz6rk"
Jul 23 11:46:17.057517 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:46:17.057458    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-8827l"]
Jul 23 11:46:17.062177 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:46:17.062142    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-8827l"
Jul 23 11:46:17.065115 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:46:17.065088    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Jul 23 11:46:17.066153 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:46:17.066132    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Jul 23 11:46:17.066276 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:46:17.066212    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\""
Jul 23 11:46:17.066339 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:46:17.066295    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-btpzj\""
Jul 23 11:46:17.069189 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:46:17.069156    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-8827l"]
Jul 23 11:46:17.146841 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:46:17.146802    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:46:17.162899 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:46:17.162862    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a431e4db-9a2b-47fe-94ff-4b92aed0b270-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-8827l\" (UID: \"a431e4db-9a2b-47fe-94ff-4b92aed0b270\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-8827l"
Jul 23 11:46:17.163054 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:46:17.162907    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a431e4db-9a2b-47fe-94ff-4b92aed0b270-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-8827l\" (UID: \"a431e4db-9a2b-47fe-94ff-4b92aed0b270\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-8827l"
Jul 23 11:46:17.163054 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:46:17.162938    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a431e4db-9a2b-47fe-94ff-4b92aed0b270-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-8827l\" (UID: \"a431e4db-9a2b-47fe-94ff-4b92aed0b270\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-8827l"
Jul 23 11:46:17.163124 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:46:17.163063    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a431e4db-9a2b-47fe-94ff-4b92aed0b270-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-8827l\" (UID: \"a431e4db-9a2b-47fe-94ff-4b92aed0b270\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-8827l"
Jul 23 11:46:17.163162 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:46:17.163131    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cssk7\" (UniqueName: \"kubernetes.io/projected/a431e4db-9a2b-47fe-94ff-4b92aed0b270-kube-api-access-cssk7\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-8827l\" (UID: \"a431e4db-9a2b-47fe-94ff-4b92aed0b270\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-8827l"
Jul 23 11:46:17.163196 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:46:17.163168    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a431e4db-9a2b-47fe-94ff-4b92aed0b270-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-8827l\" (UID: \"a431e4db-9a2b-47fe-94ff-4b92aed0b270\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-8827l"
Jul 23 11:46:17.264133 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:46:17.264095    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cssk7\" (UniqueName: \"kubernetes.io/projected/a431e4db-9a2b-47fe-94ff-4b92aed0b270-kube-api-access-cssk7\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-8827l\" (UID: \"a431e4db-9a2b-47fe-94ff-4b92aed0b270\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-8827l"
Jul 23 11:46:17.264336 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:46:17.264143    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a431e4db-9a2b-47fe-94ff-4b92aed0b270-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-8827l\" (UID: \"a431e4db-9a2b-47fe-94ff-4b92aed0b270\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-8827l"
Jul 23 11:46:17.264336 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:46:17.264169    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a431e4db-9a2b-47fe-94ff-4b92aed0b270-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-8827l\" (UID: \"a431e4db-9a2b-47fe-94ff-4b92aed0b270\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-8827l"
Jul 23 11:46:17.264336 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:46:17.264196    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a431e4db-9a2b-47fe-94ff-4b92aed0b270-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-8827l\" (UID: \"a431e4db-9a2b-47fe-94ff-4b92aed0b270\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-8827l"
Jul 23 11:46:17.264336 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:46:17.264216    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a431e4db-9a2b-47fe-94ff-4b92aed0b270-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-8827l\" (UID: \"a431e4db-9a2b-47fe-94ff-4b92aed0b270\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-8827l"
Jul 23 11:46:17.264336 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:46:17.264270    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a431e4db-9a2b-47fe-94ff-4b92aed0b270-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-8827l\" (UID: \"a431e4db-9a2b-47fe-94ff-4b92aed0b270\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-8827l"
Jul 23 11:46:17.264676 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:46:17.264643    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a431e4db-9a2b-47fe-94ff-4b92aed0b270-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-8827l\" (UID: \"a431e4db-9a2b-47fe-94ff-4b92aed0b270\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-8827l"
Jul 23 11:46:17.264676 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:46:17.264665    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a431e4db-9a2b-47fe-94ff-4b92aed0b270-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-8827l\" (UID: \"a431e4db-9a2b-47fe-94ff-4b92aed0b270\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-8827l"
Jul 23 11:46:17.264872 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:46:17.264809    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a431e4db-9a2b-47fe-94ff-4b92aed0b270-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-8827l\" (UID: \"a431e4db-9a2b-47fe-94ff-4b92aed0b270\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-8827l"
Jul 23 11:46:17.266642 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:46:17.266620    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a431e4db-9a2b-47fe-94ff-4b92aed0b270-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-8827l\" (UID: \"a431e4db-9a2b-47fe-94ff-4b92aed0b270\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-8827l"
Jul 23 11:46:17.266925 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:46:17.266906    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a431e4db-9a2b-47fe-94ff-4b92aed0b270-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-8827l\" (UID: \"a431e4db-9a2b-47fe-94ff-4b92aed0b270\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-8827l"
Jul 23 11:46:17.273537 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:46:17.273503    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cssk7\" (UniqueName: \"kubernetes.io/projected/a431e4db-9a2b-47fe-94ff-4b92aed0b270-kube-api-access-cssk7\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-8827l\" (UID: \"a431e4db-9a2b-47fe-94ff-4b92aed0b270\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-8827l"
Jul 23 11:46:17.375892 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:46:17.375847    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-8827l"
Jul 23 11:46:17.539821 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:46:17.539782    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-8827l"]
Jul 23 11:46:17.542465 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:46:17.542429    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda431e4db_9a2b_47fe_94ff_4b92aed0b270.slice/crio-6a405c1fc92899eb1946ab75133053d22bb0dcb7a3da48382426f30edfca8c06 WatchSource:0}: Error finding container 6a405c1fc92899eb1946ab75133053d22bb0dcb7a3da48382426f30edfca8c06: Status 404 returned error can't find the container with id 6a405c1fc92899eb1946ab75133053d22bb0dcb7a3da48382426f30edfca8c06
Jul 23 11:46:17.964342 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:46:17.964292    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-8827l" event={"ID":"a431e4db-9a2b-47fe-94ff-4b92aed0b270","Type":"ContainerStarted","Data":"6a405c1fc92899eb1946ab75133053d22bb0dcb7a3da48382426f30edfca8c06"}
Jul 23 11:46:18.971088 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:46:18.971047    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-8827l" event={"ID":"a431e4db-9a2b-47fe-94ff-4b92aed0b270","Type":"ContainerStarted","Data":"3c467c1c79c6094e4844d9c0a128b26d2f044258d5b21f974c4a9646f918255b"}
Jul 23 11:46:18.990199 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:46:18.990134    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-8827l" podStartSLOduration=0.724726657 podStartE2EDuration="1.990111087s" totalImagesPullingTime="1.26538443s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 11:46:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:46:18.988408349 +0000 UTC m=+633.381346138" watchObservedRunningTime="2026-07-23 11:46:18.990111087 +0000 UTC m=+633.383048875"
Jul 23 11:46:23.897546 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:46:23.897499    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:46:27.376074 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:46:27.376002    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-8827l"
Jul 23 11:46:27.376660 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:46:27.376106    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-8827l"
Jul 23 11:46:27.390169 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:46:27.390132    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-8827l"
Jul 23 11:46:28.024801 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:46:28.024769    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-8827l"
Jul 23 11:46:32.639839 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:46:32.639801    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-gfcpc"]
Jul 23 11:46:32.644934 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:46:32.644907    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gfcpc"
Jul 23 11:46:32.647544 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:46:32.647521    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\""
Jul 23 11:46:32.654793 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:46:32.654763    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-gfcpc"]
Jul 23 11:46:32.667594 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:46:32.667557    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:46:32.805840 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:46:32.805795    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20e400ab-125c-4414-b5c1-deb1455a5871-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-gfcpc\" (UID: \"20e400ab-125c-4414-b5c1-deb1455a5871\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gfcpc"
Jul 23 11:46:32.805840 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:46:32.805843    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/20e400ab-125c-4414-b5c1-deb1455a5871-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-gfcpc\" (UID: \"20e400ab-125c-4414-b5c1-deb1455a5871\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gfcpc"
Jul 23 11:46:32.806078 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:46:32.805905    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/20e400ab-125c-4414-b5c1-deb1455a5871-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-gfcpc\" (UID: \"20e400ab-125c-4414-b5c1-deb1455a5871\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gfcpc"
Jul 23 11:46:32.806078 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:46:32.805963    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20e400ab-125c-4414-b5c1-deb1455a5871-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-gfcpc\" (UID: \"20e400ab-125c-4414-b5c1-deb1455a5871\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gfcpc"
Jul 23 11:46:32.806078 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:46:32.805982    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20e400ab-125c-4414-b5c1-deb1455a5871-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-gfcpc\" (UID: \"20e400ab-125c-4414-b5c1-deb1455a5871\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gfcpc"
Jul 23 11:46:32.806222 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:46:32.806145    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5z7tw\" (UniqueName: \"kubernetes.io/projected/20e400ab-125c-4414-b5c1-deb1455a5871-kube-api-access-5z7tw\") pod \"e2e-distinct-simulated-kserve-59975d69f9-gfcpc\" (UID: \"20e400ab-125c-4414-b5c1-deb1455a5871\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gfcpc"
Jul 23 11:46:32.906870 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:46:32.906772    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5z7tw\" (UniqueName: \"kubernetes.io/projected/20e400ab-125c-4414-b5c1-deb1455a5871-kube-api-access-5z7tw\") pod \"e2e-distinct-simulated-kserve-59975d69f9-gfcpc\" (UID: \"20e400ab-125c-4414-b5c1-deb1455a5871\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gfcpc"
Jul 23 11:46:32.906870 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:46:32.906821    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20e400ab-125c-4414-b5c1-deb1455a5871-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-gfcpc\" (UID: \"20e400ab-125c-4414-b5c1-deb1455a5871\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gfcpc"
Jul 23 11:46:32.906870 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:46:32.906847    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/20e400ab-125c-4414-b5c1-deb1455a5871-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-gfcpc\" (UID: \"20e400ab-125c-4414-b5c1-deb1455a5871\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gfcpc"
Jul 23 11:46:32.906870 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:46:32.906869    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/20e400ab-125c-4414-b5c1-deb1455a5871-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-gfcpc\" (UID: \"20e400ab-125c-4414-b5c1-deb1455a5871\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gfcpc"
Jul 23 11:46:32.907187 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:46:32.906899    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20e400ab-125c-4414-b5c1-deb1455a5871-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-gfcpc\" (UID: \"20e400ab-125c-4414-b5c1-deb1455a5871\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gfcpc"
Jul 23 11:46:32.907187 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:46:32.906913    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20e400ab-125c-4414-b5c1-deb1455a5871-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-gfcpc\" (UID: \"20e400ab-125c-4414-b5c1-deb1455a5871\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gfcpc"
Jul 23 11:46:32.907385 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:46:32.907362    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20e400ab-125c-4414-b5c1-deb1455a5871-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-gfcpc\" (UID: \"20e400ab-125c-4414-b5c1-deb1455a5871\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gfcpc"
Jul 23 11:46:32.907458 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:46:32.907386    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20e400ab-125c-4414-b5c1-deb1455a5871-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-gfcpc\" (UID: \"20e400ab-125c-4414-b5c1-deb1455a5871\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gfcpc"
Jul 23 11:46:32.907458 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:46:32.907423    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/20e400ab-125c-4414-b5c1-deb1455a5871-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-gfcpc\" (UID: \"20e400ab-125c-4414-b5c1-deb1455a5871\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gfcpc"
Jul 23 11:46:32.909258 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:46:32.909236    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/20e400ab-125c-4414-b5c1-deb1455a5871-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-gfcpc\" (UID: \"20e400ab-125c-4414-b5c1-deb1455a5871\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gfcpc"
Jul 23 11:46:32.909557 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:46:32.909537    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20e400ab-125c-4414-b5c1-deb1455a5871-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-gfcpc\" (UID: \"20e400ab-125c-4414-b5c1-deb1455a5871\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gfcpc"
Jul 23 11:46:32.916055 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:46:32.916024    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5z7tw\" (UniqueName: \"kubernetes.io/projected/20e400ab-125c-4414-b5c1-deb1455a5871-kube-api-access-5z7tw\") pod \"e2e-distinct-simulated-kserve-59975d69f9-gfcpc\" (UID: \"20e400ab-125c-4414-b5c1-deb1455a5871\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gfcpc"
Jul 23 11:46:32.955883 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:46:32.955845    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gfcpc"
Jul 23 11:46:33.097378 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:46:33.097349    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-gfcpc"]
Jul 23 11:46:33.100229 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:46:33.100197    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20e400ab_125c_4414_b5c1_deb1455a5871.slice/crio-28955b0ad417f251e2a171705cced3f89ad16c8e71f129945b694cba2f097a52 WatchSource:0}: Error finding container 28955b0ad417f251e2a171705cced3f89ad16c8e71f129945b694cba2f097a52: Status 404 returned error can't find the container with id 28955b0ad417f251e2a171705cced3f89ad16c8e71f129945b694cba2f097a52
Jul 23 11:46:34.041915 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:46:34.041877    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gfcpc" event={"ID":"20e400ab-125c-4414-b5c1-deb1455a5871","Type":"ContainerStarted","Data":"0ca3b8f62297ab57e4211eb82635fe60237c1f9d18d4984498e314a014909f90"}
Jul 23 11:46:34.042313 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:46:34.041936    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gfcpc" event={"ID":"20e400ab-125c-4414-b5c1-deb1455a5871","Type":"ContainerStarted","Data":"28955b0ad417f251e2a171705cced3f89ad16c8e71f129945b694cba2f097a52"}
Jul 23 11:46:34.062332 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:46:34.062273    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gfcpc" podStartSLOduration=1.86081962 podStartE2EDuration="2.062256483s" totalImagesPullingTime="201.436863ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 11:46:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:46:34.061160513 +0000 UTC m=+648.454098306" watchObservedRunningTime="2026-07-23 11:46:34.062256483 +0000 UTC m=+648.455194370"
Jul 23 11:46:38.659720 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:46:38.659659    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:46:42.956221 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:46:42.956181    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gfcpc"
Jul 23 11:46:42.956652 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:46:42.956249    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gfcpc"
Jul 23 11:46:42.968958 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:46:42.968928    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gfcpc"
Jul 23 11:46:43.101449 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:46:43.101418    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gfcpc"
Jul 23 11:46:43.510650 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:46:43.510604    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:46:43.574802 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:46:43.574755    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:46:47.872480 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:46:47.872440    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:47:00.825336 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:47:00.825301    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-f2pzs"]
Jul 23 11:47:00.831201 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:47:00.831177    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-f2pzs"
Jul 23 11:47:00.833840 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:47:00.833810    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\""
Jul 23 11:47:00.840990 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:47:00.840962    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-f2pzs"]
Jul 23 11:47:00.858944 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:47:00.858898    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:47:00.973577 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:47:00.973535    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/906a2b44-6eee-47d1-9f8c-6657926fd9be-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-f2pzs\" (UID: \"906a2b44-6eee-47d1-9f8c-6657926fd9be\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-f2pzs"
Jul 23 11:47:00.973800 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:47:00.973589    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/906a2b44-6eee-47d1-9f8c-6657926fd9be-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-f2pzs\" (UID: \"906a2b44-6eee-47d1-9f8c-6657926fd9be\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-f2pzs"
Jul 23 11:47:00.973800 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:47:00.973616    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xx2kb\" (UniqueName: \"kubernetes.io/projected/906a2b44-6eee-47d1-9f8c-6657926fd9be-kube-api-access-xx2kb\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-f2pzs\" (UID: \"906a2b44-6eee-47d1-9f8c-6657926fd9be\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-f2pzs"
Jul 23 11:47:00.973800 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:47:00.973649    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/906a2b44-6eee-47d1-9f8c-6657926fd9be-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-f2pzs\" (UID: \"906a2b44-6eee-47d1-9f8c-6657926fd9be\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-f2pzs"
Jul 23 11:47:00.973800 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:47:00.973750    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/906a2b44-6eee-47d1-9f8c-6657926fd9be-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-f2pzs\" (UID: \"906a2b44-6eee-47d1-9f8c-6657926fd9be\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-f2pzs"
Jul 23 11:47:00.973800 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:47:00.973771    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/906a2b44-6eee-47d1-9f8c-6657926fd9be-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-f2pzs\" (UID: \"906a2b44-6eee-47d1-9f8c-6657926fd9be\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-f2pzs"
Jul 23 11:47:01.074606 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:47:01.074551    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/906a2b44-6eee-47d1-9f8c-6657926fd9be-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-f2pzs\" (UID: \"906a2b44-6eee-47d1-9f8c-6657926fd9be\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-f2pzs"
Jul 23 11:47:01.074606 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:47:01.074600    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xx2kb\" (UniqueName: \"kubernetes.io/projected/906a2b44-6eee-47d1-9f8c-6657926fd9be-kube-api-access-xx2kb\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-f2pzs\" (UID: \"906a2b44-6eee-47d1-9f8c-6657926fd9be\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-f2pzs"
Jul 23 11:47:01.074944 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:47:01.074631    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/906a2b44-6eee-47d1-9f8c-6657926fd9be-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-f2pzs\" (UID: \"906a2b44-6eee-47d1-9f8c-6657926fd9be\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-f2pzs"
Jul 23 11:47:01.074944 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:47:01.074735    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/906a2b44-6eee-47d1-9f8c-6657926fd9be-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-f2pzs\" (UID: \"906a2b44-6eee-47d1-9f8c-6657926fd9be\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-f2pzs"
Jul 23 11:47:01.074944 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:47:01.074756    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/906a2b44-6eee-47d1-9f8c-6657926fd9be-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-f2pzs\" (UID: \"906a2b44-6eee-47d1-9f8c-6657926fd9be\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-f2pzs"
Jul 23 11:47:01.074944 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:47:01.074837    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/906a2b44-6eee-47d1-9f8c-6657926fd9be-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-f2pzs\" (UID: \"906a2b44-6eee-47d1-9f8c-6657926fd9be\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-f2pzs"
Jul 23 11:47:01.075261 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:47:01.075232    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/906a2b44-6eee-47d1-9f8c-6657926fd9be-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-f2pzs\" (UID: \"906a2b44-6eee-47d1-9f8c-6657926fd9be\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-f2pzs"
Jul 23 11:47:01.075395 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:47:01.075324    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/906a2b44-6eee-47d1-9f8c-6657926fd9be-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-f2pzs\" (UID: \"906a2b44-6eee-47d1-9f8c-6657926fd9be\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-f2pzs"
Jul 23 11:47:01.075395 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:47:01.075338    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/906a2b44-6eee-47d1-9f8c-6657926fd9be-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-f2pzs\" (UID: \"906a2b44-6eee-47d1-9f8c-6657926fd9be\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-f2pzs"
Jul 23 11:47:01.077637 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:47:01.077611    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/906a2b44-6eee-47d1-9f8c-6657926fd9be-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-f2pzs\" (UID: \"906a2b44-6eee-47d1-9f8c-6657926fd9be\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-f2pzs"
Jul 23 11:47:01.077879 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:47:01.077860    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/906a2b44-6eee-47d1-9f8c-6657926fd9be-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-f2pzs\" (UID: \"906a2b44-6eee-47d1-9f8c-6657926fd9be\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-f2pzs"
Jul 23 11:47:01.084255 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:47:01.084213    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xx2kb\" (UniqueName: \"kubernetes.io/projected/906a2b44-6eee-47d1-9f8c-6657926fd9be-kube-api-access-xx2kb\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-f2pzs\" (UID: \"906a2b44-6eee-47d1-9f8c-6657926fd9be\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-f2pzs"
Jul 23 11:47:01.143343 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:47:01.143305    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-f2pzs"
Jul 23 11:47:01.286832 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:47:01.286805    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-f2pzs"]
Jul 23 11:47:01.289214 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:47:01.289187    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod906a2b44_6eee_47d1_9f8c_6657926fd9be.slice/crio-18b2ebad9358ad922227e8b2e85e017744eb7e33fe5d1e8f4a24f054abc1608a WatchSource:0}: Error finding container 18b2ebad9358ad922227e8b2e85e017744eb7e33fe5d1e8f4a24f054abc1608a: Status 404 returned error can't find the container with id 18b2ebad9358ad922227e8b2e85e017744eb7e33fe5d1e8f4a24f054abc1608a
Jul 23 11:47:02.178941 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:47:02.178899    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-f2pzs" event={"ID":"906a2b44-6eee-47d1-9f8c-6657926fd9be","Type":"ContainerStarted","Data":"5ce7fdf8b4f9cd63107a68c0801437ce03d614b87c50da4c728f29f48008c20c"}
Jul 23 11:47:02.179360 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:47:02.178957    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-f2pzs" event={"ID":"906a2b44-6eee-47d1-9f8c-6657926fd9be","Type":"ContainerStarted","Data":"18b2ebad9358ad922227e8b2e85e017744eb7e33fe5d1e8f4a24f054abc1608a"}
Jul 23 11:47:02.199465 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:47:02.199407    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-f2pzs" podStartSLOduration=1.758183866 podStartE2EDuration="2.199389094s" totalImagesPullingTime="441.205228ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 11:47:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:47:02.198350736 +0000 UTC m=+676.591288525" watchObservedRunningTime="2026-07-23 11:47:02.199389094 +0000 UTC m=+676.592326881"
Jul 23 11:47:07.075035 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:47:07.074993    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:47:07.134913 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:47:07.134867    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:47:11.143885 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:47:11.143841    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-f2pzs"
Jul 23 11:47:11.144290 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:47:11.144089    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-f2pzs"
Jul 23 11:47:11.156877 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:47:11.156844    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-f2pzs"
Jul 23 11:47:11.241844 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:47:11.241810    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-f2pzs"
Jul 23 11:47:14.338756 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:47:14.338722    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-x9l6l"]
Jul 23 11:47:14.342163 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:47:14.342137    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-x9l6l"
Jul 23 11:47:14.345647 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:47:14.345621    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Jul 23 11:47:14.356958 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:47:14.356917    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-x9l6l"]
Jul 23 11:47:14.412829 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:47:14.412792    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:47:14.501289 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:47:14.501253    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e0e9242c-b071-4bf2-aa3e-1f176d0f5ae5-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-x9l6l\" (UID: \"e0e9242c-b071-4bf2-aa3e-1f176d0f5ae5\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-x9l6l"
Jul 23 11:47:14.501485 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:47:14.501314    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mvntm\" (UniqueName: \"kubernetes.io/projected/e0e9242c-b071-4bf2-aa3e-1f176d0f5ae5-kube-api-access-mvntm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-x9l6l\" (UID: \"e0e9242c-b071-4bf2-aa3e-1f176d0f5ae5\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-x9l6l"
Jul 23 11:47:14.501485 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:47:14.501371    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0e9242c-b071-4bf2-aa3e-1f176d0f5ae5-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-x9l6l\" (UID: \"e0e9242c-b071-4bf2-aa3e-1f176d0f5ae5\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-x9l6l"
Jul 23 11:47:14.501485 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:47:14.501427    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e0e9242c-b071-4bf2-aa3e-1f176d0f5ae5-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-x9l6l\" (UID: \"e0e9242c-b071-4bf2-aa3e-1f176d0f5ae5\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-x9l6l"
Jul 23 11:47:14.501485 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:47:14.501476    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e0e9242c-b071-4bf2-aa3e-1f176d0f5ae5-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-x9l6l\" (UID: \"e0e9242c-b071-4bf2-aa3e-1f176d0f5ae5\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-x9l6l"
Jul 23 11:47:14.501627 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:47:14.501525    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e0e9242c-b071-4bf2-aa3e-1f176d0f5ae5-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-x9l6l\" (UID: \"e0e9242c-b071-4bf2-aa3e-1f176d0f5ae5\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-x9l6l"
Jul 23 11:47:14.602258 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:47:14.602160    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e0e9242c-b071-4bf2-aa3e-1f176d0f5ae5-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-x9l6l\" (UID: \"e0e9242c-b071-4bf2-aa3e-1f176d0f5ae5\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-x9l6l"
Jul 23 11:47:14.602258 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:47:14.602236    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e0e9242c-b071-4bf2-aa3e-1f176d0f5ae5-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-x9l6l\" (UID: \"e0e9242c-b071-4bf2-aa3e-1f176d0f5ae5\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-x9l6l"
Jul 23 11:47:14.602477 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:47:14.602270    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvntm\" (UniqueName: \"kubernetes.io/projected/e0e9242c-b071-4bf2-aa3e-1f176d0f5ae5-kube-api-access-mvntm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-x9l6l\" (UID: \"e0e9242c-b071-4bf2-aa3e-1f176d0f5ae5\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-x9l6l"
Jul 23 11:47:14.602477 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:47:14.602292    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0e9242c-b071-4bf2-aa3e-1f176d0f5ae5-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-x9l6l\" (UID: \"e0e9242c-b071-4bf2-aa3e-1f176d0f5ae5\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-x9l6l"
Jul 23 11:47:14.602477 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:47:14.602330    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e0e9242c-b071-4bf2-aa3e-1f176d0f5ae5-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-x9l6l\" (UID: \"e0e9242c-b071-4bf2-aa3e-1f176d0f5ae5\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-x9l6l"
Jul 23 11:47:14.602477 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:47:14.602387    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e0e9242c-b071-4bf2-aa3e-1f176d0f5ae5-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-x9l6l\" (UID: \"e0e9242c-b071-4bf2-aa3e-1f176d0f5ae5\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-x9l6l"
Jul 23 11:47:14.602739 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:47:14.602714    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e0e9242c-b071-4bf2-aa3e-1f176d0f5ae5-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-x9l6l\" (UID: \"e0e9242c-b071-4bf2-aa3e-1f176d0f5ae5\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-x9l6l"
Jul 23 11:47:14.602832 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:47:14.602773    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0e9242c-b071-4bf2-aa3e-1f176d0f5ae5-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-x9l6l\" (UID: \"e0e9242c-b071-4bf2-aa3e-1f176d0f5ae5\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-x9l6l"
Jul 23 11:47:14.602832 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:47:14.602817    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e0e9242c-b071-4bf2-aa3e-1f176d0f5ae5-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-x9l6l\" (UID: \"e0e9242c-b071-4bf2-aa3e-1f176d0f5ae5\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-x9l6l"
Jul 23 11:47:14.604850 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:47:14.604814    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e0e9242c-b071-4bf2-aa3e-1f176d0f5ae5-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-x9l6l\" (UID: \"e0e9242c-b071-4bf2-aa3e-1f176d0f5ae5\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-x9l6l"
Jul 23 11:47:14.605083 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:47:14.605066    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e0e9242c-b071-4bf2-aa3e-1f176d0f5ae5-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-x9l6l\" (UID: \"e0e9242c-b071-4bf2-aa3e-1f176d0f5ae5\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-x9l6l"
Jul 23 11:47:14.610739 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:47:14.610716    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvntm\" (UniqueName: \"kubernetes.io/projected/e0e9242c-b071-4bf2-aa3e-1f176d0f5ae5-kube-api-access-mvntm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-x9l6l\" (UID: \"e0e9242c-b071-4bf2-aa3e-1f176d0f5ae5\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-x9l6l"
Jul 23 11:47:14.652736 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:47:14.652665    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-x9l6l"
Jul 23 11:47:14.801387 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:47:14.801355    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-x9l6l"]
Jul 23 11:47:14.803909 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:47:14.803882    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode0e9242c_b071_4bf2_aa3e_1f176d0f5ae5.slice/crio-1db42d4d21123ba82bc0aca4c8b1492faabf5ab140ba39b6acb0ffbbe98f1a3d WatchSource:0}: Error finding container 1db42d4d21123ba82bc0aca4c8b1492faabf5ab140ba39b6acb0ffbbe98f1a3d: Status 404 returned error can't find the container with id 1db42d4d21123ba82bc0aca4c8b1492faabf5ab140ba39b6acb0ffbbe98f1a3d
Jul 23 11:47:15.251003 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:47:15.250967    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-x9l6l" event={"ID":"e0e9242c-b071-4bf2-aa3e-1f176d0f5ae5","Type":"ContainerStarted","Data":"1db42d4d21123ba82bc0aca4c8b1492faabf5ab140ba39b6acb0ffbbe98f1a3d"}
Jul 23 11:47:16.256969 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:47:16.256927    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-x9l6l" event={"ID":"e0e9242c-b071-4bf2-aa3e-1f176d0f5ae5","Type":"ContainerStarted","Data":"f6672ae7b652798ea1d8a743da8b4a79e5fa13a274d020bb7951c74c271e3db2"}
Jul 23 11:47:16.277223 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:47:16.277162    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-x9l6l" podStartSLOduration=1.343880209 podStartE2EDuration="2.277144749s" totalImagesPullingTime="933.26454ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 11:47:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:47:16.275884114 +0000 UTC m=+690.668821906" watchObservedRunningTime="2026-07-23 11:47:16.277144749 +0000 UTC m=+690.670082536"
Jul 23 11:47:24.653776 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:47:24.653719    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-x9l6l"
Jul 23 11:47:24.653776 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:47:24.653788    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-x9l6l"
Jul 23 11:47:24.666799 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:47:24.666769    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-x9l6l"
Jul 23 11:47:25.322994 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:47:25.322960    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-x9l6l"
Jul 23 11:47:27.900205 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:47:27.900167    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:48:21.274712 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:48:21.274649    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:48:21.372905 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:48:21.372858    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:48:22.503921 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:48:22.503878    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:48:22.602695 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:48:22.602657    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:49:08.982312 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:49:08.982229    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-78746d79b5-w9j78"]
Jul 23 11:49:08.982923 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:49:08.982432    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-78746d79b5-w9j78" podUID="c0c873c3-6ff4-4f3f-a81f-0d3b1fb87307" containerName="manager" containerID="cri-o://02245d0c8e15d36befbb783cede1a59535212ee27fbf441e49b9e7fded7244de" gracePeriod=30
Jul 23 11:49:09.235353 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:49:09.235280    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-78746d79b5-w9j78"
Jul 23 11:49:09.370709 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:49:09.370633    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c0c873c3-6ff4-4f3f-a81f-0d3b1fb87307-webhook-cert\") pod \"c0c873c3-6ff4-4f3f-a81f-0d3b1fb87307\" (UID: \"c0c873c3-6ff4-4f3f-a81f-0d3b1fb87307\") "
Jul 23 11:49:09.370899 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:49:09.370739    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z4qzw\" (UniqueName: \"kubernetes.io/projected/c0c873c3-6ff4-4f3f-a81f-0d3b1fb87307-kube-api-access-z4qzw\") pod \"c0c873c3-6ff4-4f3f-a81f-0d3b1fb87307\" (UID: \"c0c873c3-6ff4-4f3f-a81f-0d3b1fb87307\") "
Jul 23 11:49:09.373039 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:49:09.373001    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0c873c3-6ff4-4f3f-a81f-0d3b1fb87307-kube-api-access-z4qzw" (OuterVolumeSpecName: "kube-api-access-z4qzw") pod "c0c873c3-6ff4-4f3f-a81f-0d3b1fb87307" (UID: "c0c873c3-6ff4-4f3f-a81f-0d3b1fb87307"). InnerVolumeSpecName "kube-api-access-z4qzw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 11:49:09.373039 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:49:09.373002    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0c873c3-6ff4-4f3f-a81f-0d3b1fb87307-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "c0c873c3-6ff4-4f3f-a81f-0d3b1fb87307" (UID: "c0c873c3-6ff4-4f3f-a81f-0d3b1fb87307"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 11:49:09.471935 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:49:09.471901    2563 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c0c873c3-6ff4-4f3f-a81f-0d3b1fb87307-webhook-cert\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 23 11:49:09.471935 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:49:09.471933    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z4qzw\" (UniqueName: \"kubernetes.io/projected/c0c873c3-6ff4-4f3f-a81f-0d3b1fb87307-kube-api-access-z4qzw\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 23 11:49:09.804325 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:49:09.804283    2563 generic.go:356] "Generic (PLEG): container finished" podID="c0c873c3-6ff4-4f3f-a81f-0d3b1fb87307" containerID="02245d0c8e15d36befbb783cede1a59535212ee27fbf441e49b9e7fded7244de" exitCode=0
Jul 23 11:49:09.804513 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:49:09.804331    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-78746d79b5-w9j78"
Jul 23 11:49:09.804513 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:49:09.804337    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-78746d79b5-w9j78" event={"ID":"c0c873c3-6ff4-4f3f-a81f-0d3b1fb87307","Type":"ContainerDied","Data":"02245d0c8e15d36befbb783cede1a59535212ee27fbf441e49b9e7fded7244de"}
Jul 23 11:49:09.804513 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:49:09.804382    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-78746d79b5-w9j78" event={"ID":"c0c873c3-6ff4-4f3f-a81f-0d3b1fb87307","Type":"ContainerDied","Data":"e59a9c49209d4903ad35fe0da1f4cc011c40dbbf7859d59d71715632fa33d31b"}
Jul 23 11:49:09.804513 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:49:09.804406    2563 scope.go:117] "RemoveContainer" containerID="02245d0c8e15d36befbb783cede1a59535212ee27fbf441e49b9e7fded7244de"
Jul 23 11:49:09.813879 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:49:09.813859    2563 scope.go:117] "RemoveContainer" containerID="02245d0c8e15d36befbb783cede1a59535212ee27fbf441e49b9e7fded7244de"
Jul 23 11:49:09.814167 ip-10-0-141-229 kubenswrapper[2563]: E0723 11:49:09.814148    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"02245d0c8e15d36befbb783cede1a59535212ee27fbf441e49b9e7fded7244de\": container with ID starting with 02245d0c8e15d36befbb783cede1a59535212ee27fbf441e49b9e7fded7244de not found: ID does not exist" containerID="02245d0c8e15d36befbb783cede1a59535212ee27fbf441e49b9e7fded7244de"
Jul 23 11:49:09.814216 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:49:09.814176    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02245d0c8e15d36befbb783cede1a59535212ee27fbf441e49b9e7fded7244de"} err="failed to get container status \"02245d0c8e15d36befbb783cede1a59535212ee27fbf441e49b9e7fded7244de\": rpc error: code = NotFound desc = could not find container \"02245d0c8e15d36befbb783cede1a59535212ee27fbf441e49b9e7fded7244de\": container with ID starting with 02245d0c8e15d36befbb783cede1a59535212ee27fbf441e49b9e7fded7244de not found: ID does not exist"
Jul 23 11:49:09.826470 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:49:09.826443    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-78746d79b5-w9j78"]
Jul 23 11:49:09.831456 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:49:09.831430    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-78746d79b5-w9j78"]
Jul 23 11:49:10.078281 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:49:10.078200    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c0c873c3-6ff4-4f3f-a81f-0d3b1fb87307" path="/var/lib/kubelet/pods/c0c873c3-6ff4-4f3f-a81f-0d3b1fb87307/volumes"
Jul 23 11:50:10.673338 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:50:10.673300    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-78746d79b5-q5wgf"]
Jul 23 11:50:10.673768 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:50:10.673665    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c0c873c3-6ff4-4f3f-a81f-0d3b1fb87307" containerName="manager"
Jul 23 11:50:10.673768 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:50:10.673676    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="c0c873c3-6ff4-4f3f-a81f-0d3b1fb87307" containerName="manager"
Jul 23 11:50:10.673768 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:50:10.673755    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="c0c873c3-6ff4-4f3f-a81f-0d3b1fb87307" containerName="manager"
Jul 23 11:50:10.676619 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:50:10.676601    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-78746d79b5-q5wgf"
Jul 23 11:50:10.679952 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:50:10.679930    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-nwgls\""
Jul 23 11:50:10.679952 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:50:10.679939    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Jul 23 11:50:10.680118 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:50:10.679961    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Jul 23 11:50:10.688156 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:50:10.688132    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-78746d79b5-q5wgf"]
Jul 23 11:50:10.840728 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:50:10.840645    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2cd95bdd-fe47-423f-abdf-31d89314ed87-webhook-cert\") pod \"maas-controller-78746d79b5-q5wgf\" (UID: \"2cd95bdd-fe47-423f-abdf-31d89314ed87\") " pod="opendatahub/maas-controller-78746d79b5-q5wgf"
Jul 23 11:50:10.840907 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:50:10.840745    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwpk8\" (UniqueName: \"kubernetes.io/projected/2cd95bdd-fe47-423f-abdf-31d89314ed87-kube-api-access-pwpk8\") pod \"maas-controller-78746d79b5-q5wgf\" (UID: \"2cd95bdd-fe47-423f-abdf-31d89314ed87\") " pod="opendatahub/maas-controller-78746d79b5-q5wgf"
Jul 23 11:50:10.942210 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:50:10.942128    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2cd95bdd-fe47-423f-abdf-31d89314ed87-webhook-cert\") pod \"maas-controller-78746d79b5-q5wgf\" (UID: \"2cd95bdd-fe47-423f-abdf-31d89314ed87\") " pod="opendatahub/maas-controller-78746d79b5-q5wgf"
Jul 23 11:50:10.942210 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:50:10.942175    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwpk8\" (UniqueName: \"kubernetes.io/projected/2cd95bdd-fe47-423f-abdf-31d89314ed87-kube-api-access-pwpk8\") pod \"maas-controller-78746d79b5-q5wgf\" (UID: \"2cd95bdd-fe47-423f-abdf-31d89314ed87\") " pod="opendatahub/maas-controller-78746d79b5-q5wgf"
Jul 23 11:50:10.944705 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:50:10.944653    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2cd95bdd-fe47-423f-abdf-31d89314ed87-webhook-cert\") pod \"maas-controller-78746d79b5-q5wgf\" (UID: \"2cd95bdd-fe47-423f-abdf-31d89314ed87\") " pod="opendatahub/maas-controller-78746d79b5-q5wgf"
Jul 23 11:50:10.952068 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:50:10.952043    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwpk8\" (UniqueName: \"kubernetes.io/projected/2cd95bdd-fe47-423f-abdf-31d89314ed87-kube-api-access-pwpk8\") pod \"maas-controller-78746d79b5-q5wgf\" (UID: \"2cd95bdd-fe47-423f-abdf-31d89314ed87\") " pod="opendatahub/maas-controller-78746d79b5-q5wgf"
Jul 23 11:50:10.986841 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:50:10.986810    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-78746d79b5-q5wgf"
Jul 23 11:50:11.120194 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:50:11.120159    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-78746d79b5-q5wgf"]
Jul 23 11:50:11.123129 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:50:11.123103    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2cd95bdd_fe47_423f_abdf_31d89314ed87.slice/crio-4ecb36bcd1f5fe374760fc8a0a7cf691b595c79d6a850cf2555ff484d0f803b6 WatchSource:0}: Error finding container 4ecb36bcd1f5fe374760fc8a0a7cf691b595c79d6a850cf2555ff484d0f803b6: Status 404 returned error can't find the container with id 4ecb36bcd1f5fe374760fc8a0a7cf691b595c79d6a850cf2555ff484d0f803b6
Jul 23 11:50:12.090906 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:50:12.090873    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-78746d79b5-q5wgf" event={"ID":"2cd95bdd-fe47-423f-abdf-31d89314ed87","Type":"ContainerStarted","Data":"bcaf659ee75b9cf1c2cf3bbcca9fac112536031bb1153ca41318329ac78be368"}
Jul 23 11:50:12.091333 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:50:12.090931    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-78746d79b5-q5wgf" event={"ID":"2cd95bdd-fe47-423f-abdf-31d89314ed87","Type":"ContainerStarted","Data":"4ecb36bcd1f5fe374760fc8a0a7cf691b595c79d6a850cf2555ff484d0f803b6"}
Jul 23 11:50:12.091333 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:50:12.090995    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-78746d79b5-q5wgf"
Jul 23 11:50:12.110281 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:50:12.110232    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-78746d79b5-q5wgf" podStartSLOduration=1.6482500629999999 podStartE2EDuration="2.110216281s" totalImagesPullingTime="461.966218ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 11:50:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:50:12.108554388 +0000 UTC m=+866.501492176" watchObservedRunningTime="2026-07-23 11:50:12.110216281 +0000 UTC m=+866.503154269"
Jul 23 11:50:23.098237 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:50:23.098204    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-78746d79b5-q5wgf"
Jul 23 11:50:27.699899 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:50:27.699863    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:50:27.768970 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:50:27.768917    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-78746d79b5-q5wgf"]
Jul 23 11:50:27.769260 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:50:27.769224    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-78746d79b5-q5wgf" podUID="2cd95bdd-fe47-423f-abdf-31d89314ed87" containerName="manager" containerID="cri-o://bcaf659ee75b9cf1c2cf3bbcca9fac112536031bb1153ca41318329ac78be368" gracePeriod=30
Jul 23 11:50:28.115797 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:50:28.115774    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-78746d79b5-q5wgf"
Jul 23 11:50:28.170398 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:50:28.170364    2563 generic.go:356] "Generic (PLEG): container finished" podID="2cd95bdd-fe47-423f-abdf-31d89314ed87" containerID="bcaf659ee75b9cf1c2cf3bbcca9fac112536031bb1153ca41318329ac78be368" exitCode=0
Jul 23 11:50:28.170398 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:50:28.170408    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-78746d79b5-q5wgf"
Jul 23 11:50:28.170656 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:50:28.170445    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-78746d79b5-q5wgf" event={"ID":"2cd95bdd-fe47-423f-abdf-31d89314ed87","Type":"ContainerDied","Data":"bcaf659ee75b9cf1c2cf3bbcca9fac112536031bb1153ca41318329ac78be368"}
Jul 23 11:50:28.170656 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:50:28.170505    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-78746d79b5-q5wgf" event={"ID":"2cd95bdd-fe47-423f-abdf-31d89314ed87","Type":"ContainerDied","Data":"4ecb36bcd1f5fe374760fc8a0a7cf691b595c79d6a850cf2555ff484d0f803b6"}
Jul 23 11:50:28.170656 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:50:28.170540    2563 scope.go:117] "RemoveContainer" containerID="bcaf659ee75b9cf1c2cf3bbcca9fac112536031bb1153ca41318329ac78be368"
Jul 23 11:50:28.181452 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:50:28.181378    2563 scope.go:117] "RemoveContainer" containerID="bcaf659ee75b9cf1c2cf3bbcca9fac112536031bb1153ca41318329ac78be368"
Jul 23 11:50:28.182714 ip-10-0-141-229 kubenswrapper[2563]: E0723 11:50:28.182651    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bcaf659ee75b9cf1c2cf3bbcca9fac112536031bb1153ca41318329ac78be368\": container with ID starting with bcaf659ee75b9cf1c2cf3bbcca9fac112536031bb1153ca41318329ac78be368 not found: ID does not exist" containerID="bcaf659ee75b9cf1c2cf3bbcca9fac112536031bb1153ca41318329ac78be368"
Jul 23 11:50:28.182714 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:50:28.182711    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bcaf659ee75b9cf1c2cf3bbcca9fac112536031bb1153ca41318329ac78be368"} err="failed to get container status \"bcaf659ee75b9cf1c2cf3bbcca9fac112536031bb1153ca41318329ac78be368\": rpc error: code = NotFound desc = could not find container \"bcaf659ee75b9cf1c2cf3bbcca9fac112536031bb1153ca41318329ac78be368\": container with ID starting with bcaf659ee75b9cf1c2cf3bbcca9fac112536031bb1153ca41318329ac78be368 not found: ID does not exist"
Jul 23 11:50:28.194192 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:50:28.194170    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2cd95bdd-fe47-423f-abdf-31d89314ed87-webhook-cert\") pod \"2cd95bdd-fe47-423f-abdf-31d89314ed87\" (UID: \"2cd95bdd-fe47-423f-abdf-31d89314ed87\") "
Jul 23 11:50:28.194328 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:50:28.194268    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pwpk8\" (UniqueName: \"kubernetes.io/projected/2cd95bdd-fe47-423f-abdf-31d89314ed87-kube-api-access-pwpk8\") pod \"2cd95bdd-fe47-423f-abdf-31d89314ed87\" (UID: \"2cd95bdd-fe47-423f-abdf-31d89314ed87\") "
Jul 23 11:50:28.196264 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:50:28.196235    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2cd95bdd-fe47-423f-abdf-31d89314ed87-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "2cd95bdd-fe47-423f-abdf-31d89314ed87" (UID: "2cd95bdd-fe47-423f-abdf-31d89314ed87"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 11:50:28.196371 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:50:28.196282    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2cd95bdd-fe47-423f-abdf-31d89314ed87-kube-api-access-pwpk8" (OuterVolumeSpecName: "kube-api-access-pwpk8") pod "2cd95bdd-fe47-423f-abdf-31d89314ed87" (UID: "2cd95bdd-fe47-423f-abdf-31d89314ed87"). InnerVolumeSpecName "kube-api-access-pwpk8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 11:50:28.295130 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:50:28.295036    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pwpk8\" (UniqueName: \"kubernetes.io/projected/2cd95bdd-fe47-423f-abdf-31d89314ed87-kube-api-access-pwpk8\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 23 11:50:28.295130 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:50:28.295068    2563 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2cd95bdd-fe47-423f-abdf-31d89314ed87-webhook-cert\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 23 11:50:28.491985 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:50:28.491955    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-78746d79b5-q5wgf"]
Jul 23 11:50:28.497353 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:50:28.497324    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-78746d79b5-q5wgf"]
Jul 23 11:50:29.197834 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:50:29.197804    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:50:29.315901 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:50:29.315865    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-78746d79b5-rrm4t"]
Jul 23 11:50:29.316226 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:50:29.316214    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2cd95bdd-fe47-423f-abdf-31d89314ed87" containerName="manager"
Jul 23 11:50:29.316269 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:50:29.316228    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="2cd95bdd-fe47-423f-abdf-31d89314ed87" containerName="manager"
Jul 23 11:50:29.316303 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:50:29.316290    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="2cd95bdd-fe47-423f-abdf-31d89314ed87" containerName="manager"
Jul 23 11:50:29.320410 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:50:29.320393    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-78746d79b5-rrm4t"
Jul 23 11:50:29.323353 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:50:29.323334    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-nwgls\""
Jul 23 11:50:29.323481 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:50:29.323335    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Jul 23 11:50:29.323481 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:50:29.323370    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Jul 23 11:50:29.330531 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:50:29.330506    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-78746d79b5-rrm4t"]
Jul 23 11:50:29.405223 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:50:29.405184    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4f71af9a-2a14-4316-bb1f-194520cb518b-webhook-cert\") pod \"maas-controller-78746d79b5-rrm4t\" (UID: \"4f71af9a-2a14-4316-bb1f-194520cb518b\") " pod="opendatahub/maas-controller-78746d79b5-rrm4t"
Jul 23 11:50:29.405410 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:50:29.405334    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n27rr\" (UniqueName: \"kubernetes.io/projected/4f71af9a-2a14-4316-bb1f-194520cb518b-kube-api-access-n27rr\") pod \"maas-controller-78746d79b5-rrm4t\" (UID: \"4f71af9a-2a14-4316-bb1f-194520cb518b\") " pod="opendatahub/maas-controller-78746d79b5-rrm4t"
Jul 23 11:50:29.505880 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:50:29.505790    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n27rr\" (UniqueName: \"kubernetes.io/projected/4f71af9a-2a14-4316-bb1f-194520cb518b-kube-api-access-n27rr\") pod \"maas-controller-78746d79b5-rrm4t\" (UID: \"4f71af9a-2a14-4316-bb1f-194520cb518b\") " pod="opendatahub/maas-controller-78746d79b5-rrm4t"
Jul 23 11:50:29.505880 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:50:29.505845    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4f71af9a-2a14-4316-bb1f-194520cb518b-webhook-cert\") pod \"maas-controller-78746d79b5-rrm4t\" (UID: \"4f71af9a-2a14-4316-bb1f-194520cb518b\") " pod="opendatahub/maas-controller-78746d79b5-rrm4t"
Jul 23 11:50:29.508354 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:50:29.508331    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4f71af9a-2a14-4316-bb1f-194520cb518b-webhook-cert\") pod \"maas-controller-78746d79b5-rrm4t\" (UID: \"4f71af9a-2a14-4316-bb1f-194520cb518b\") " pod="opendatahub/maas-controller-78746d79b5-rrm4t"
Jul 23 11:50:29.513940 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:50:29.513920    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n27rr\" (UniqueName: \"kubernetes.io/projected/4f71af9a-2a14-4316-bb1f-194520cb518b-kube-api-access-n27rr\") pod \"maas-controller-78746d79b5-rrm4t\" (UID: \"4f71af9a-2a14-4316-bb1f-194520cb518b\") " pod="opendatahub/maas-controller-78746d79b5-rrm4t"
Jul 23 11:50:29.631504 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:50:29.631472    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-78746d79b5-rrm4t"
Jul 23 11:50:29.766914 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:50:29.766869    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-78746d79b5-rrm4t"]
Jul 23 11:50:29.769124 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:50:29.769100    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f71af9a_2a14_4316_bb1f_194520cb518b.slice/crio-24650dfb82222abe50c7d54c7ce66cc561220d67e070f6817f0ae2c637110eac WatchSource:0}: Error finding container 24650dfb82222abe50c7d54c7ce66cc561220d67e070f6817f0ae2c637110eac: Status 404 returned error can't find the container with id 24650dfb82222abe50c7d54c7ce66cc561220d67e070f6817f0ae2c637110eac
Jul 23 11:50:30.077945 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:50:30.077863    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2cd95bdd-fe47-423f-abdf-31d89314ed87" path="/var/lib/kubelet/pods/2cd95bdd-fe47-423f-abdf-31d89314ed87/volumes"
Jul 23 11:50:30.180958 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:50:30.180930    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-78746d79b5-rrm4t" event={"ID":"4f71af9a-2a14-4316-bb1f-194520cb518b","Type":"ContainerStarted","Data":"24650dfb82222abe50c7d54c7ce66cc561220d67e070f6817f0ae2c637110eac"}
Jul 23 11:50:31.186414 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:50:31.186377    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-78746d79b5-rrm4t" event={"ID":"4f71af9a-2a14-4316-bb1f-194520cb518b","Type":"ContainerStarted","Data":"99d003d06ffe6cacb203c915c35e540feac56c1c8e44bfda5e910f8f52f82675"}
Jul 23 11:50:31.186832 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:50:31.186444    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-78746d79b5-rrm4t"
Jul 23 11:50:31.204174 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:50:31.204117    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-78746d79b5-rrm4t" podStartSLOduration=1.846469408 podStartE2EDuration="2.204102447s" totalImagesPullingTime="357.633039ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 11:50:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:50:31.203334999 +0000 UTC m=+885.596272788" watchObservedRunningTime="2026-07-23 11:50:31.204102447 +0000 UTC m=+885.597040217"
Jul 23 11:50:42.193385 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:50:42.193353    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-78746d79b5-rrm4t"
Jul 23 11:50:46.020654 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:50:46.020621    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nhm9p_93649ac9-9062-46c0-adaa-1048d471211a/ovn-acl-logging/0.log"
Jul 23 11:50:46.021732 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:50:46.021704    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nhm9p_93649ac9-9062-46c0-adaa-1048d471211a/ovn-acl-logging/0.log"
Jul 23 11:50:46.787111 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:50:46.787067    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:50:47.493726 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:50:47.493662    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:51:08.722851 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:51:08.722813    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:51:10.197991 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:51:10.197954    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:51:14.857243 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:51:14.857176    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:51:14.969581 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:51:14.969542    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:51:15.059981 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:51:15.059943    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:51:15.284954 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:51:15.284918    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:51:21.036503 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:51:21.036464    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:51:21.116987 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:51:21.116955    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:53:59.523858 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:53:59.523776    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:53:59.611946 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:53:59.611916    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:55:46.051979 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:55:46.051950    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nhm9p_93649ac9-9062-46c0-adaa-1048d471211a/ovn-acl-logging/0.log"
Jul 23 11:55:46.056150 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:55:46.056124    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nhm9p_93649ac9-9062-46c0-adaa-1048d471211a/ovn-acl-logging/0.log"
Jul 23 11:57:03.928444 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:57:03.928408    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:57:05.449389 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:57:05.449348    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:59:13.151524 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:59:13.151492    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-f30565c5-kserve-6fd989fc9-8l4rl"]
Jul 23 11:59:13.155035 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:59:13.155015    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-f30565c5-kserve-6fd989fc9-8l4rl"
Jul 23 11:59:13.157672 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:59:13.157645    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-f30565c5-kserve-self-signed-certs\""
Jul 23 11:59:13.173629 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:59:13.173592    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-f30565c5-kserve-6fd989fc9-8l4rl"]
Jul 23 11:59:13.219608 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:59:13.219569    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:59:13.295259 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:59:13.295215    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ff20572f-7777-4bac-9f7f-de4aaa64fd55-home\") pod \"e2e-same-id-a-f30565c5-kserve-6fd989fc9-8l4rl\" (UID: \"ff20572f-7777-4bac-9f7f-de4aaa64fd55\") " pod="llm/e2e-same-id-a-f30565c5-kserve-6fd989fc9-8l4rl"
Jul 23 11:59:13.295432 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:59:13.295270    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ff20572f-7777-4bac-9f7f-de4aaa64fd55-dshm\") pod \"e2e-same-id-a-f30565c5-kserve-6fd989fc9-8l4rl\" (UID: \"ff20572f-7777-4bac-9f7f-de4aaa64fd55\") " pod="llm/e2e-same-id-a-f30565c5-kserve-6fd989fc9-8l4rl"
Jul 23 11:59:13.295432 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:59:13.295297    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff20572f-7777-4bac-9f7f-de4aaa64fd55-tmp-dir\") pod \"e2e-same-id-a-f30565c5-kserve-6fd989fc9-8l4rl\" (UID: \"ff20572f-7777-4bac-9f7f-de4aaa64fd55\") " pod="llm/e2e-same-id-a-f30565c5-kserve-6fd989fc9-8l4rl"
Jul 23 11:59:13.295432 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:59:13.295320    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff20572f-7777-4bac-9f7f-de4aaa64fd55-tls-certs\") pod \"e2e-same-id-a-f30565c5-kserve-6fd989fc9-8l4rl\" (UID: \"ff20572f-7777-4bac-9f7f-de4aaa64fd55\") " pod="llm/e2e-same-id-a-f30565c5-kserve-6fd989fc9-8l4rl"
Jul 23 11:59:13.295432 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:59:13.295338    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ff20572f-7777-4bac-9f7f-de4aaa64fd55-model-cache\") pod \"e2e-same-id-a-f30565c5-kserve-6fd989fc9-8l4rl\" (UID: \"ff20572f-7777-4bac-9f7f-de4aaa64fd55\") " pod="llm/e2e-same-id-a-f30565c5-kserve-6fd989fc9-8l4rl"
Jul 23 11:59:13.295432 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:59:13.295375    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4xvr6\" (UniqueName: \"kubernetes.io/projected/ff20572f-7777-4bac-9f7f-de4aaa64fd55-kube-api-access-4xvr6\") pod \"e2e-same-id-a-f30565c5-kserve-6fd989fc9-8l4rl\" (UID: \"ff20572f-7777-4bac-9f7f-de4aaa64fd55\") " pod="llm/e2e-same-id-a-f30565c5-kserve-6fd989fc9-8l4rl"
Jul 23 11:59:13.396776 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:59:13.396731    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ff20572f-7777-4bac-9f7f-de4aaa64fd55-home\") pod \"e2e-same-id-a-f30565c5-kserve-6fd989fc9-8l4rl\" (UID: \"ff20572f-7777-4bac-9f7f-de4aaa64fd55\") " pod="llm/e2e-same-id-a-f30565c5-kserve-6fd989fc9-8l4rl"
Jul 23 11:59:13.396984 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:59:13.396798    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ff20572f-7777-4bac-9f7f-de4aaa64fd55-dshm\") pod \"e2e-same-id-a-f30565c5-kserve-6fd989fc9-8l4rl\" (UID: \"ff20572f-7777-4bac-9f7f-de4aaa64fd55\") " pod="llm/e2e-same-id-a-f30565c5-kserve-6fd989fc9-8l4rl"
Jul 23 11:59:13.396984 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:59:13.396821    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff20572f-7777-4bac-9f7f-de4aaa64fd55-tmp-dir\") pod \"e2e-same-id-a-f30565c5-kserve-6fd989fc9-8l4rl\" (UID: \"ff20572f-7777-4bac-9f7f-de4aaa64fd55\") " pod="llm/e2e-same-id-a-f30565c5-kserve-6fd989fc9-8l4rl"
Jul 23 11:59:13.396984 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:59:13.396847    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff20572f-7777-4bac-9f7f-de4aaa64fd55-tls-certs\") pod \"e2e-same-id-a-f30565c5-kserve-6fd989fc9-8l4rl\" (UID: \"ff20572f-7777-4bac-9f7f-de4aaa64fd55\") " pod="llm/e2e-same-id-a-f30565c5-kserve-6fd989fc9-8l4rl"
Jul 23 11:59:13.396984 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:59:13.396865    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ff20572f-7777-4bac-9f7f-de4aaa64fd55-model-cache\") pod \"e2e-same-id-a-f30565c5-kserve-6fd989fc9-8l4rl\" (UID: \"ff20572f-7777-4bac-9f7f-de4aaa64fd55\") " pod="llm/e2e-same-id-a-f30565c5-kserve-6fd989fc9-8l4rl"
Jul 23 11:59:13.396984 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:59:13.396887    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xvr6\" (UniqueName: \"kubernetes.io/projected/ff20572f-7777-4bac-9f7f-de4aaa64fd55-kube-api-access-4xvr6\") pod \"e2e-same-id-a-f30565c5-kserve-6fd989fc9-8l4rl\" (UID: \"ff20572f-7777-4bac-9f7f-de4aaa64fd55\") " pod="llm/e2e-same-id-a-f30565c5-kserve-6fd989fc9-8l4rl"
Jul 23 11:59:13.397261 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:59:13.397211    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ff20572f-7777-4bac-9f7f-de4aaa64fd55-home\") pod \"e2e-same-id-a-f30565c5-kserve-6fd989fc9-8l4rl\" (UID: \"ff20572f-7777-4bac-9f7f-de4aaa64fd55\") " pod="llm/e2e-same-id-a-f30565c5-kserve-6fd989fc9-8l4rl"
Jul 23 11:59:13.397323 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:59:13.397297    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff20572f-7777-4bac-9f7f-de4aaa64fd55-tmp-dir\") pod \"e2e-same-id-a-f30565c5-kserve-6fd989fc9-8l4rl\" (UID: \"ff20572f-7777-4bac-9f7f-de4aaa64fd55\") " pod="llm/e2e-same-id-a-f30565c5-kserve-6fd989fc9-8l4rl"
Jul 23 11:59:13.397366 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:59:13.397333    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ff20572f-7777-4bac-9f7f-de4aaa64fd55-model-cache\") pod \"e2e-same-id-a-f30565c5-kserve-6fd989fc9-8l4rl\" (UID: \"ff20572f-7777-4bac-9f7f-de4aaa64fd55\") " pod="llm/e2e-same-id-a-f30565c5-kserve-6fd989fc9-8l4rl"
Jul 23 11:59:13.399373 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:59:13.399346    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ff20572f-7777-4bac-9f7f-de4aaa64fd55-dshm\") pod \"e2e-same-id-a-f30565c5-kserve-6fd989fc9-8l4rl\" (UID: \"ff20572f-7777-4bac-9f7f-de4aaa64fd55\") " pod="llm/e2e-same-id-a-f30565c5-kserve-6fd989fc9-8l4rl"
Jul 23 11:59:13.399551 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:59:13.399532    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff20572f-7777-4bac-9f7f-de4aaa64fd55-tls-certs\") pod \"e2e-same-id-a-f30565c5-kserve-6fd989fc9-8l4rl\" (UID: \"ff20572f-7777-4bac-9f7f-de4aaa64fd55\") " pod="llm/e2e-same-id-a-f30565c5-kserve-6fd989fc9-8l4rl"
Jul 23 11:59:13.408046 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:59:13.407966    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xvr6\" (UniqueName: \"kubernetes.io/projected/ff20572f-7777-4bac-9f7f-de4aaa64fd55-kube-api-access-4xvr6\") pod \"e2e-same-id-a-f30565c5-kserve-6fd989fc9-8l4rl\" (UID: \"ff20572f-7777-4bac-9f7f-de4aaa64fd55\") " pod="llm/e2e-same-id-a-f30565c5-kserve-6fd989fc9-8l4rl"
Jul 23 11:59:13.463875 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:59:13.463824    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-f30565c5-kserve-6fd989fc9-8l4rl"
Jul 23 11:59:13.611700 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:59:13.611653    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-f30565c5-kserve-6fd989fc9-8l4rl"]
Jul 23 11:59:13.614437 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:59:13.614404    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podff20572f_7777_4bac_9f7f_de4aaa64fd55.slice/crio-697b4dca48bb04e04d1ca98c7f375594460176cf3397baa2d24fce13685f65de WatchSource:0}: Error finding container 697b4dca48bb04e04d1ca98c7f375594460176cf3397baa2d24fce13685f65de: Status 404 returned error can't find the container with id 697b4dca48bb04e04d1ca98c7f375594460176cf3397baa2d24fce13685f65de
Jul 23 11:59:13.632326 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:59:13.632286    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-f30565c5-kserve-6fd989fc9-8l4rl" event={"ID":"ff20572f-7777-4bac-9f7f-de4aaa64fd55","Type":"ContainerStarted","Data":"697b4dca48bb04e04d1ca98c7f375594460176cf3397baa2d24fce13685f65de"}
Jul 23 11:59:14.009605 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:59:14.009512    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:59:14.638391 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:59:14.638350    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-f30565c5-kserve-6fd989fc9-8l4rl" event={"ID":"ff20572f-7777-4bac-9f7f-de4aaa64fd55","Type":"ContainerStarted","Data":"a82aaece51422a73ae55eb7802805e4c645e94e1b47eb99df8b74b95559909a0"}
Jul 23 11:59:14.657701 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:59:14.657618    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-f30565c5-kserve-6fd989fc9-8l4rl" podStartSLOduration=1.657594572 podStartE2EDuration="1.657594572s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 11:59:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:59:14.65530613 +0000 UTC m=+1409.048243919" watchObservedRunningTime="2026-07-23 11:59:14.657594572 +0000 UTC m=+1409.050532362"
Jul 23 11:59:22.334154 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:59:22.334118    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-f30565c5-kserve-599c5c9d6c-2bb2w"]
Jul 23 11:59:22.337776 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:59:22.337747    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-f30565c5-kserve-599c5c9d6c-2bb2w"
Jul 23 11:59:22.340480 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:59:22.340455    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-f30565c5-kserve-self-signed-certs\""
Jul 23 11:59:22.351901 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:59:22.351869    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-f30565c5-kserve-599c5c9d6c-2bb2w"]
Jul 23 11:59:22.380575 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:59:22.380535    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:59:22.384982 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:59:22.384944    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa4e467e-fcf7-4d22-9170-96a0a856a991-home\") pod \"e2e-same-id-b-f30565c5-kserve-599c5c9d6c-2bb2w\" (UID: \"fa4e467e-fcf7-4d22-9170-96a0a856a991\") " pod="llm/e2e-same-id-b-f30565c5-kserve-599c5c9d6c-2bb2w"
Jul 23 11:59:22.385126 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:59:22.384988    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa4e467e-fcf7-4d22-9170-96a0a856a991-dshm\") pod \"e2e-same-id-b-f30565c5-kserve-599c5c9d6c-2bb2w\" (UID: \"fa4e467e-fcf7-4d22-9170-96a0a856a991\") " pod="llm/e2e-same-id-b-f30565c5-kserve-599c5c9d6c-2bb2w"
Jul 23 11:59:22.385126 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:59:22.385009    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vdsrg\" (UniqueName: \"kubernetes.io/projected/fa4e467e-fcf7-4d22-9170-96a0a856a991-kube-api-access-vdsrg\") pod \"e2e-same-id-b-f30565c5-kserve-599c5c9d6c-2bb2w\" (UID: \"fa4e467e-fcf7-4d22-9170-96a0a856a991\") " pod="llm/e2e-same-id-b-f30565c5-kserve-599c5c9d6c-2bb2w"
Jul 23 11:59:22.385202 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:59:22.385125    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa4e467e-fcf7-4d22-9170-96a0a856a991-model-cache\") pod \"e2e-same-id-b-f30565c5-kserve-599c5c9d6c-2bb2w\" (UID: \"fa4e467e-fcf7-4d22-9170-96a0a856a991\") " pod="llm/e2e-same-id-b-f30565c5-kserve-599c5c9d6c-2bb2w"
Jul 23 11:59:22.385202 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:59:22.385170    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa4e467e-fcf7-4d22-9170-96a0a856a991-tls-certs\") pod \"e2e-same-id-b-f30565c5-kserve-599c5c9d6c-2bb2w\" (UID: \"fa4e467e-fcf7-4d22-9170-96a0a856a991\") " pod="llm/e2e-same-id-b-f30565c5-kserve-599c5c9d6c-2bb2w"
Jul 23 11:59:22.385270 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:59:22.385242    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa4e467e-fcf7-4d22-9170-96a0a856a991-tmp-dir\") pod \"e2e-same-id-b-f30565c5-kserve-599c5c9d6c-2bb2w\" (UID: \"fa4e467e-fcf7-4d22-9170-96a0a856a991\") " pod="llm/e2e-same-id-b-f30565c5-kserve-599c5c9d6c-2bb2w"
Jul 23 11:59:22.486126 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:59:22.486089    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa4e467e-fcf7-4d22-9170-96a0a856a991-model-cache\") pod \"e2e-same-id-b-f30565c5-kserve-599c5c9d6c-2bb2w\" (UID: \"fa4e467e-fcf7-4d22-9170-96a0a856a991\") " pod="llm/e2e-same-id-b-f30565c5-kserve-599c5c9d6c-2bb2w"
Jul 23 11:59:22.486319 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:59:22.486143    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa4e467e-fcf7-4d22-9170-96a0a856a991-tls-certs\") pod \"e2e-same-id-b-f30565c5-kserve-599c5c9d6c-2bb2w\" (UID: \"fa4e467e-fcf7-4d22-9170-96a0a856a991\") " pod="llm/e2e-same-id-b-f30565c5-kserve-599c5c9d6c-2bb2w"
Jul 23 11:59:22.486319 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:59:22.486207    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa4e467e-fcf7-4d22-9170-96a0a856a991-tmp-dir\") pod \"e2e-same-id-b-f30565c5-kserve-599c5c9d6c-2bb2w\" (UID: \"fa4e467e-fcf7-4d22-9170-96a0a856a991\") " pod="llm/e2e-same-id-b-f30565c5-kserve-599c5c9d6c-2bb2w"
Jul 23 11:59:22.486319 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:59:22.486236    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa4e467e-fcf7-4d22-9170-96a0a856a991-home\") pod \"e2e-same-id-b-f30565c5-kserve-599c5c9d6c-2bb2w\" (UID: \"fa4e467e-fcf7-4d22-9170-96a0a856a991\") " pod="llm/e2e-same-id-b-f30565c5-kserve-599c5c9d6c-2bb2w"
Jul 23 11:59:22.486319 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:59:22.486258    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa4e467e-fcf7-4d22-9170-96a0a856a991-dshm\") pod \"e2e-same-id-b-f30565c5-kserve-599c5c9d6c-2bb2w\" (UID: \"fa4e467e-fcf7-4d22-9170-96a0a856a991\") " pod="llm/e2e-same-id-b-f30565c5-kserve-599c5c9d6c-2bb2w"
Jul 23 11:59:22.486319 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:59:22.486280    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdsrg\" (UniqueName: \"kubernetes.io/projected/fa4e467e-fcf7-4d22-9170-96a0a856a991-kube-api-access-vdsrg\") pod \"e2e-same-id-b-f30565c5-kserve-599c5c9d6c-2bb2w\" (UID: \"fa4e467e-fcf7-4d22-9170-96a0a856a991\") " pod="llm/e2e-same-id-b-f30565c5-kserve-599c5c9d6c-2bb2w"
Jul 23 11:59:22.486582 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:59:22.486555    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa4e467e-fcf7-4d22-9170-96a0a856a991-model-cache\") pod \"e2e-same-id-b-f30565c5-kserve-599c5c9d6c-2bb2w\" (UID: \"fa4e467e-fcf7-4d22-9170-96a0a856a991\") " pod="llm/e2e-same-id-b-f30565c5-kserve-599c5c9d6c-2bb2w"
Jul 23 11:59:22.486640 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:59:22.486605    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa4e467e-fcf7-4d22-9170-96a0a856a991-tmp-dir\") pod \"e2e-same-id-b-f30565c5-kserve-599c5c9d6c-2bb2w\" (UID: \"fa4e467e-fcf7-4d22-9170-96a0a856a991\") " pod="llm/e2e-same-id-b-f30565c5-kserve-599c5c9d6c-2bb2w"
Jul 23 11:59:22.486724 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:59:22.486631    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa4e467e-fcf7-4d22-9170-96a0a856a991-home\") pod \"e2e-same-id-b-f30565c5-kserve-599c5c9d6c-2bb2w\" (UID: \"fa4e467e-fcf7-4d22-9170-96a0a856a991\") " pod="llm/e2e-same-id-b-f30565c5-kserve-599c5c9d6c-2bb2w"
Jul 23 11:59:22.488717 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:59:22.488674    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa4e467e-fcf7-4d22-9170-96a0a856a991-dshm\") pod \"e2e-same-id-b-f30565c5-kserve-599c5c9d6c-2bb2w\" (UID: \"fa4e467e-fcf7-4d22-9170-96a0a856a991\") " pod="llm/e2e-same-id-b-f30565c5-kserve-599c5c9d6c-2bb2w"
Jul 23 11:59:22.488865 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:59:22.488847    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa4e467e-fcf7-4d22-9170-96a0a856a991-tls-certs\") pod \"e2e-same-id-b-f30565c5-kserve-599c5c9d6c-2bb2w\" (UID: \"fa4e467e-fcf7-4d22-9170-96a0a856a991\") " pod="llm/e2e-same-id-b-f30565c5-kserve-599c5c9d6c-2bb2w"
Jul 23 11:59:22.495710 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:59:22.495649    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdsrg\" (UniqueName: \"kubernetes.io/projected/fa4e467e-fcf7-4d22-9170-96a0a856a991-kube-api-access-vdsrg\") pod \"e2e-same-id-b-f30565c5-kserve-599c5c9d6c-2bb2w\" (UID: \"fa4e467e-fcf7-4d22-9170-96a0a856a991\") " pod="llm/e2e-same-id-b-f30565c5-kserve-599c5c9d6c-2bb2w"
Jul 23 11:59:22.647544 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:59:22.647505    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-f30565c5-kserve-599c5c9d6c-2bb2w"
Jul 23 11:59:22.789768 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:59:22.789514    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-f30565c5-kserve-599c5c9d6c-2bb2w"]
Jul 23 11:59:22.792107 ip-10-0-141-229 kubenswrapper[2563]: W0723 11:59:22.792077    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfa4e467e_fcf7_4d22_9170_96a0a856a991.slice/crio-dfdda2273318d07af109ad1243460228abb436c868024a424d2120f4b468e152 WatchSource:0}: Error finding container dfdda2273318d07af109ad1243460228abb436c868024a424d2120f4b468e152: Status 404 returned error can't find the container with id dfdda2273318d07af109ad1243460228abb436c868024a424d2120f4b468e152
Jul 23 11:59:23.464217 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:59:23.464113    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-f30565c5-kserve-6fd989fc9-8l4rl"
Jul 23 11:59:23.464217 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:59:23.464164    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-f30565c5-kserve-6fd989fc9-8l4rl"
Jul 23 11:59:23.477151 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:59:23.477121    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-f30565c5-kserve-6fd989fc9-8l4rl"
Jul 23 11:59:23.684334 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:59:23.684296    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-f30565c5-kserve-599c5c9d6c-2bb2w" event={"ID":"fa4e467e-fcf7-4d22-9170-96a0a856a991","Type":"ContainerStarted","Data":"fe805ca369e03584c32cc6e7f9ff08fb70b659cc6243e8bfd22b7c256b283fb3"}
Jul 23 11:59:23.684553 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:59:23.684347    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-f30565c5-kserve-599c5c9d6c-2bb2w" event={"ID":"fa4e467e-fcf7-4d22-9170-96a0a856a991","Type":"ContainerStarted","Data":"dfdda2273318d07af109ad1243460228abb436c868024a424d2120f4b468e152"}
Jul 23 11:59:23.696104 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:59:23.696072    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-f30565c5-kserve-6fd989fc9-8l4rl"
Jul 23 11:59:23.701165 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:59:23.701118    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-f30565c5-kserve-599c5c9d6c-2bb2w" podStartSLOduration=1.701102075 podStartE2EDuration="1.701102075s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 11:59:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:59:23.700045199 +0000 UTC m=+1418.092982984" watchObservedRunningTime="2026-07-23 11:59:23.701102075 +0000 UTC m=+1418.094039896"
Jul 23 11:59:32.648200 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:59:32.648143    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-f30565c5-kserve-599c5c9d6c-2bb2w"
Jul 23 11:59:32.648626 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:59:32.648211    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-f30565c5-kserve-599c5c9d6c-2bb2w"
Jul 23 11:59:32.661149 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:59:32.661115    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-f30565c5-kserve-599c5c9d6c-2bb2w"
Jul 23 11:59:32.738609 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:59:32.738574    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-f30565c5-kserve-599c5c9d6c-2bb2w"
Jul 23 11:59:51.111310 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:59:51.111224    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-f30565c5-kserve-6fd989fc9-8l4rl"]
Jul 23 11:59:51.134131 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:59:51.134096    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 11:59:51.767158 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:59:51.767113    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-f30565c5-kserve-599c5c9d6c-2bb2w"]
Jul 23 11:59:51.797807 ip-10-0-141-229 kubenswrapper[2563]: I0723 11:59:51.797766    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:00:06.133205 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:00:06.133120    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-f30565c5-kserve-6fd989fc9-8l4rl" podUID="ff20572f-7777-4bac-9f7f-de4aaa64fd55" containerName="main" containerID="cri-o://a82aaece51422a73ae55eb7802805e4c645e94e1b47eb99df8b74b95559909a0" gracePeriod=45
Jul 23 12:00:06.388194 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:00:06.388106    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-f30565c5-kserve-6fd989fc9-8l4rl"
Jul 23 12:00:06.511999 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:00:06.511955    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff20572f-7777-4bac-9f7f-de4aaa64fd55-tmp-dir\") pod \"ff20572f-7777-4bac-9f7f-de4aaa64fd55\" (UID: \"ff20572f-7777-4bac-9f7f-de4aaa64fd55\") "
Jul 23 12:00:06.512196 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:00:06.512023    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ff20572f-7777-4bac-9f7f-de4aaa64fd55-model-cache\") pod \"ff20572f-7777-4bac-9f7f-de4aaa64fd55\" (UID: \"ff20572f-7777-4bac-9f7f-de4aaa64fd55\") "
Jul 23 12:00:06.512196 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:00:06.512046    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4xvr6\" (UniqueName: \"kubernetes.io/projected/ff20572f-7777-4bac-9f7f-de4aaa64fd55-kube-api-access-4xvr6\") pod \"ff20572f-7777-4bac-9f7f-de4aaa64fd55\" (UID: \"ff20572f-7777-4bac-9f7f-de4aaa64fd55\") "
Jul 23 12:00:06.512196 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:00:06.512079    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff20572f-7777-4bac-9f7f-de4aaa64fd55-tls-certs\") pod \"ff20572f-7777-4bac-9f7f-de4aaa64fd55\" (UID: \"ff20572f-7777-4bac-9f7f-de4aaa64fd55\") "
Jul 23 12:00:06.512196 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:00:06.512103    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ff20572f-7777-4bac-9f7f-de4aaa64fd55-home\") pod \"ff20572f-7777-4bac-9f7f-de4aaa64fd55\" (UID: \"ff20572f-7777-4bac-9f7f-de4aaa64fd55\") "
Jul 23 12:00:06.512196 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:00:06.512124    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ff20572f-7777-4bac-9f7f-de4aaa64fd55-dshm\") pod \"ff20572f-7777-4bac-9f7f-de4aaa64fd55\" (UID: \"ff20572f-7777-4bac-9f7f-de4aaa64fd55\") "
Jul 23 12:00:06.512468 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:00:06.512314    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff20572f-7777-4bac-9f7f-de4aaa64fd55-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ff20572f-7777-4bac-9f7f-de4aaa64fd55" (UID: "ff20572f-7777-4bac-9f7f-de4aaa64fd55"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 12:00:06.512468 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:00:06.512398    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff20572f-7777-4bac-9f7f-de4aaa64fd55-home" (OuterVolumeSpecName: "home") pod "ff20572f-7777-4bac-9f7f-de4aaa64fd55" (UID: "ff20572f-7777-4bac-9f7f-de4aaa64fd55"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 12:00:06.512468 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:00:06.512324    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff20572f-7777-4bac-9f7f-de4aaa64fd55-model-cache" (OuterVolumeSpecName: "model-cache") pod "ff20572f-7777-4bac-9f7f-de4aaa64fd55" (UID: "ff20572f-7777-4bac-9f7f-de4aaa64fd55"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 12:00:06.512604 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:00:06.512509    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff20572f-7777-4bac-9f7f-de4aaa64fd55-tmp-dir\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 23 12:00:06.512604 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:00:06.512530    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ff20572f-7777-4bac-9f7f-de4aaa64fd55-model-cache\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 23 12:00:06.512604 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:00:06.512548    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ff20572f-7777-4bac-9f7f-de4aaa64fd55-home\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 23 12:00:06.514674 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:00:06.514639    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff20572f-7777-4bac-9f7f-de4aaa64fd55-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ff20572f-7777-4bac-9f7f-de4aaa64fd55" (UID: "ff20572f-7777-4bac-9f7f-de4aaa64fd55"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 12:00:06.514840 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:00:06.514704    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff20572f-7777-4bac-9f7f-de4aaa64fd55-dshm" (OuterVolumeSpecName: "dshm") pod "ff20572f-7777-4bac-9f7f-de4aaa64fd55" (UID: "ff20572f-7777-4bac-9f7f-de4aaa64fd55"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 12:00:06.514840 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:00:06.514781    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff20572f-7777-4bac-9f7f-de4aaa64fd55-kube-api-access-4xvr6" (OuterVolumeSpecName: "kube-api-access-4xvr6") pod "ff20572f-7777-4bac-9f7f-de4aaa64fd55" (UID: "ff20572f-7777-4bac-9f7f-de4aaa64fd55"). InnerVolumeSpecName "kube-api-access-4xvr6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 12:00:06.614030 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:00:06.613978    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ff20572f-7777-4bac-9f7f-de4aaa64fd55-dshm\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 23 12:00:06.614030 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:00:06.614016    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4xvr6\" (UniqueName: \"kubernetes.io/projected/ff20572f-7777-4bac-9f7f-de4aaa64fd55-kube-api-access-4xvr6\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 23 12:00:06.614030 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:00:06.614027    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff20572f-7777-4bac-9f7f-de4aaa64fd55-tls-certs\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 23 12:00:06.789832 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:00:06.789711    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-f30565c5-kserve-599c5c9d6c-2bb2w" podUID="fa4e467e-fcf7-4d22-9170-96a0a856a991" containerName="main" containerID="cri-o://fe805ca369e03584c32cc6e7f9ff08fb70b659cc6243e8bfd22b7c256b283fb3" gracePeriod=45
Jul 23 12:00:06.892798 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:00:06.892745    2563 generic.go:356] "Generic (PLEG): container finished" podID="ff20572f-7777-4bac-9f7f-de4aaa64fd55" containerID="a82aaece51422a73ae55eb7802805e4c645e94e1b47eb99df8b74b95559909a0" exitCode=0
Jul 23 12:00:06.893026 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:00:06.892808    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-f30565c5-kserve-6fd989fc9-8l4rl"
Jul 23 12:00:06.893026 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:00:06.892808    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-f30565c5-kserve-6fd989fc9-8l4rl" event={"ID":"ff20572f-7777-4bac-9f7f-de4aaa64fd55","Type":"ContainerDied","Data":"a82aaece51422a73ae55eb7802805e4c645e94e1b47eb99df8b74b95559909a0"}
Jul 23 12:00:06.893026 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:00:06.892869    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-f30565c5-kserve-6fd989fc9-8l4rl" event={"ID":"ff20572f-7777-4bac-9f7f-de4aaa64fd55","Type":"ContainerDied","Data":"697b4dca48bb04e04d1ca98c7f375594460176cf3397baa2d24fce13685f65de"}
Jul 23 12:00:06.893026 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:00:06.892898    2563 scope.go:117] "RemoveContainer" containerID="a82aaece51422a73ae55eb7802805e4c645e94e1b47eb99df8b74b95559909a0"
Jul 23 12:00:06.906505 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:00:06.906482    2563 scope.go:117] "RemoveContainer" containerID="a82aaece51422a73ae55eb7802805e4c645e94e1b47eb99df8b74b95559909a0"
Jul 23 12:00:06.906936 ip-10-0-141-229 kubenswrapper[2563]: E0723 12:00:06.906914    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a82aaece51422a73ae55eb7802805e4c645e94e1b47eb99df8b74b95559909a0\": container with ID starting with a82aaece51422a73ae55eb7802805e4c645e94e1b47eb99df8b74b95559909a0 not found: ID does not exist" containerID="a82aaece51422a73ae55eb7802805e4c645e94e1b47eb99df8b74b95559909a0"
Jul 23 12:00:06.907014 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:00:06.906948    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a82aaece51422a73ae55eb7802805e4c645e94e1b47eb99df8b74b95559909a0"} err="failed to get container status \"a82aaece51422a73ae55eb7802805e4c645e94e1b47eb99df8b74b95559909a0\": rpc error: code = NotFound desc = could not find container \"a82aaece51422a73ae55eb7802805e4c645e94e1b47eb99df8b74b95559909a0\": container with ID starting with a82aaece51422a73ae55eb7802805e4c645e94e1b47eb99df8b74b95559909a0 not found: ID does not exist"
Jul 23 12:00:06.919578 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:00:06.919531    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-f30565c5-kserve-6fd989fc9-8l4rl"]
Jul 23 12:00:06.924205 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:00:06.924168    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-f30565c5-kserve-6fd989fc9-8l4rl"]
Jul 23 12:00:07.051280 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:00:07.051248    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-f30565c5-kserve-599c5c9d6c-2bb2w"
Jul 23 12:00:07.119465 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:00:07.119424    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vdsrg\" (UniqueName: \"kubernetes.io/projected/fa4e467e-fcf7-4d22-9170-96a0a856a991-kube-api-access-vdsrg\") pod \"fa4e467e-fcf7-4d22-9170-96a0a856a991\" (UID: \"fa4e467e-fcf7-4d22-9170-96a0a856a991\") "
Jul 23 12:00:07.119465 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:00:07.119471    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa4e467e-fcf7-4d22-9170-96a0a856a991-dshm\") pod \"fa4e467e-fcf7-4d22-9170-96a0a856a991\" (UID: \"fa4e467e-fcf7-4d22-9170-96a0a856a991\") "
Jul 23 12:00:07.119761 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:00:07.119492    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa4e467e-fcf7-4d22-9170-96a0a856a991-tls-certs\") pod \"fa4e467e-fcf7-4d22-9170-96a0a856a991\" (UID: \"fa4e467e-fcf7-4d22-9170-96a0a856a991\") "
Jul 23 12:00:07.119761 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:00:07.119525    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa4e467e-fcf7-4d22-9170-96a0a856a991-model-cache\") pod \"fa4e467e-fcf7-4d22-9170-96a0a856a991\" (UID: \"fa4e467e-fcf7-4d22-9170-96a0a856a991\") "
Jul 23 12:00:07.119761 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:00:07.119654    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa4e467e-fcf7-4d22-9170-96a0a856a991-home\") pod \"fa4e467e-fcf7-4d22-9170-96a0a856a991\" (UID: \"fa4e467e-fcf7-4d22-9170-96a0a856a991\") "
Jul 23 12:00:07.119761 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:00:07.119716    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa4e467e-fcf7-4d22-9170-96a0a856a991-tmp-dir\") pod \"fa4e467e-fcf7-4d22-9170-96a0a856a991\" (UID: \"fa4e467e-fcf7-4d22-9170-96a0a856a991\") "
Jul 23 12:00:07.119974 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:00:07.119846    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa4e467e-fcf7-4d22-9170-96a0a856a991-model-cache" (OuterVolumeSpecName: "model-cache") pod "fa4e467e-fcf7-4d22-9170-96a0a856a991" (UID: "fa4e467e-fcf7-4d22-9170-96a0a856a991"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 12:00:07.120033 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:00:07.119959    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa4e467e-fcf7-4d22-9170-96a0a856a991-home" (OuterVolumeSpecName: "home") pod "fa4e467e-fcf7-4d22-9170-96a0a856a991" (UID: "fa4e467e-fcf7-4d22-9170-96a0a856a991"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 12:00:07.120033 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:00:07.120003    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa4e467e-fcf7-4d22-9170-96a0a856a991-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "fa4e467e-fcf7-4d22-9170-96a0a856a991" (UID: "fa4e467e-fcf7-4d22-9170-96a0a856a991"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 12:00:07.120119 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:00:07.120103    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa4e467e-fcf7-4d22-9170-96a0a856a991-model-cache\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 23 12:00:07.120154 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:00:07.120123    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa4e467e-fcf7-4d22-9170-96a0a856a991-home\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 23 12:00:07.120154 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:00:07.120136    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa4e467e-fcf7-4d22-9170-96a0a856a991-tmp-dir\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 23 12:00:07.121980 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:00:07.121956    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa4e467e-fcf7-4d22-9170-96a0a856a991-dshm" (OuterVolumeSpecName: "dshm") pod "fa4e467e-fcf7-4d22-9170-96a0a856a991" (UID: "fa4e467e-fcf7-4d22-9170-96a0a856a991"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 12:00:07.122101 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:00:07.121992    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa4e467e-fcf7-4d22-9170-96a0a856a991-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "fa4e467e-fcf7-4d22-9170-96a0a856a991" (UID: "fa4e467e-fcf7-4d22-9170-96a0a856a991"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 12:00:07.122101 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:00:07.122086    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa4e467e-fcf7-4d22-9170-96a0a856a991-kube-api-access-vdsrg" (OuterVolumeSpecName: "kube-api-access-vdsrg") pod "fa4e467e-fcf7-4d22-9170-96a0a856a991" (UID: "fa4e467e-fcf7-4d22-9170-96a0a856a991"). InnerVolumeSpecName "kube-api-access-vdsrg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 12:00:07.220829 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:00:07.220785    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vdsrg\" (UniqueName: \"kubernetes.io/projected/fa4e467e-fcf7-4d22-9170-96a0a856a991-kube-api-access-vdsrg\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 23 12:00:07.220829 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:00:07.220820    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa4e467e-fcf7-4d22-9170-96a0a856a991-dshm\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 23 12:00:07.220829 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:00:07.220830    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa4e467e-fcf7-4d22-9170-96a0a856a991-tls-certs\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 23 12:00:07.899194 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:00:07.899156    2563 generic.go:356] "Generic (PLEG): container finished" podID="fa4e467e-fcf7-4d22-9170-96a0a856a991" containerID="fe805ca369e03584c32cc6e7f9ff08fb70b659cc6243e8bfd22b7c256b283fb3" exitCode=0
Jul 23 12:00:07.899395 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:00:07.899203    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-f30565c5-kserve-599c5c9d6c-2bb2w"
Jul 23 12:00:07.899395 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:00:07.899245    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-f30565c5-kserve-599c5c9d6c-2bb2w" event={"ID":"fa4e467e-fcf7-4d22-9170-96a0a856a991","Type":"ContainerDied","Data":"fe805ca369e03584c32cc6e7f9ff08fb70b659cc6243e8bfd22b7c256b283fb3"}
Jul 23 12:00:07.899395 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:00:07.899288    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-f30565c5-kserve-599c5c9d6c-2bb2w" event={"ID":"fa4e467e-fcf7-4d22-9170-96a0a856a991","Type":"ContainerDied","Data":"dfdda2273318d07af109ad1243460228abb436c868024a424d2120f4b468e152"}
Jul 23 12:00:07.899395 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:00:07.899312    2563 scope.go:117] "RemoveContainer" containerID="fe805ca369e03584c32cc6e7f9ff08fb70b659cc6243e8bfd22b7c256b283fb3"
Jul 23 12:00:07.909190 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:00:07.909166    2563 scope.go:117] "RemoveContainer" containerID="fe805ca369e03584c32cc6e7f9ff08fb70b659cc6243e8bfd22b7c256b283fb3"
Jul 23 12:00:07.909562 ip-10-0-141-229 kubenswrapper[2563]: E0723 12:00:07.909540    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe805ca369e03584c32cc6e7f9ff08fb70b659cc6243e8bfd22b7c256b283fb3\": container with ID starting with fe805ca369e03584c32cc6e7f9ff08fb70b659cc6243e8bfd22b7c256b283fb3 not found: ID does not exist" containerID="fe805ca369e03584c32cc6e7f9ff08fb70b659cc6243e8bfd22b7c256b283fb3"
Jul 23 12:00:07.909649 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:00:07.909573    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe805ca369e03584c32cc6e7f9ff08fb70b659cc6243e8bfd22b7c256b283fb3"} err="failed to get container status \"fe805ca369e03584c32cc6e7f9ff08fb70b659cc6243e8bfd22b7c256b283fb3\": rpc error: code = NotFound desc = could not find container \"fe805ca369e03584c32cc6e7f9ff08fb70b659cc6243e8bfd22b7c256b283fb3\": container with ID starting with fe805ca369e03584c32cc6e7f9ff08fb70b659cc6243e8bfd22b7c256b283fb3 not found: ID does not exist"
Jul 23 12:00:07.922289 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:00:07.922239    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-f30565c5-kserve-599c5c9d6c-2bb2w"]
Jul 23 12:00:07.927671 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:00:07.927635    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-f30565c5-kserve-599c5c9d6c-2bb2w"]
Jul 23 12:00:08.078087 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:00:08.078052    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fa4e467e-fcf7-4d22-9170-96a0a856a991" path="/var/lib/kubelet/pods/fa4e467e-fcf7-4d22-9170-96a0a856a991/volumes"
Jul 23 12:00:08.078466 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:00:08.078452    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ff20572f-7777-4bac-9f7f-de4aaa64fd55" path="/var/lib/kubelet/pods/ff20572f-7777-4bac-9f7f-de4aaa64fd55/volumes"
Jul 23 12:00:22.374264 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:00:22.374230    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:00:22.473035 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:00:22.472990    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:00:38.541746 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:00:38.541702    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:00:46.085532 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:00:46.085504    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nhm9p_93649ac9-9062-46c0-adaa-1048d471211a/ovn-acl-logging/0.log"
Jul 23 12:00:46.090075 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:00:46.090052    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nhm9p_93649ac9-9062-46c0-adaa-1048d471211a/ovn-acl-logging/0.log"
Jul 23 12:03:11.435830 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:03:11.435743    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:03:33.753215 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:03:33.753175    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:03:36.297873 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:03:36.297843    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:03:36.395040 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:03:36.395004    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:03:36.798506 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:03:36.798475    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:03:43.492664 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:03:43.492628    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:03:43.591727 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:03:43.591668    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:03:43.731958 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:03:43.731926    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:03:44.295579 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:03:44.295542    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:03:45.072213 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:03:45.072175    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:03:46.285183 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:03:46.285150    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:03:46.394095 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:03:46.394039    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:03:46.406721 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:03:46.406672    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-9fab72d8-59c4fd9fd8-8gb4m"]
Jul 23 12:03:46.407130 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:03:46.407112    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa4e467e-fcf7-4d22-9170-96a0a856a991" containerName="main"
Jul 23 12:03:46.407208 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:03:46.407133    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa4e467e-fcf7-4d22-9170-96a0a856a991" containerName="main"
Jul 23 12:03:46.407208 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:03:46.407148    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff20572f-7777-4bac-9f7f-de4aaa64fd55" containerName="main"
Jul 23 12:03:46.407208 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:03:46.407156    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff20572f-7777-4bac-9f7f-de4aaa64fd55" containerName="main"
Jul 23 12:03:46.407365 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:03:46.407252    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="ff20572f-7777-4bac-9f7f-de4aaa64fd55" containerName="main"
Jul 23 12:03:46.407365 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:03:46.407263    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="fa4e467e-fcf7-4d22-9170-96a0a856a991" containerName="main"
Jul 23 12:03:46.410120 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:03:46.410098    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9fab72d8-59c4fd9fd8-8gb4m"
Jul 23 12:03:46.412576 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:03:46.412557    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-9fab72d8-serving-cert\""
Jul 23 12:03:46.433361 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:03:46.433330    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-9fab72d8-59c4fd9fd8-8gb4m"]
Jul 23 12:03:46.464184 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:03:46.464135    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rm54g\" (UniqueName: \"kubernetes.io/projected/e38019f0-946a-402f-978f-b09315e01913-kube-api-access-rm54g\") pod \"maas-api-e2e-ait-9fab72d8-59c4fd9fd8-8gb4m\" (UID: \"e38019f0-946a-402f-978f-b09315e01913\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9fab72d8-59c4fd9fd8-8gb4m"
Jul 23 12:03:46.464373 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:03:46.464217    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e38019f0-946a-402f-978f-b09315e01913-maas-api-tls\") pod \"maas-api-e2e-ait-9fab72d8-59c4fd9fd8-8gb4m\" (UID: \"e38019f0-946a-402f-978f-b09315e01913\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9fab72d8-59c4fd9fd8-8gb4m"
Jul 23 12:03:46.464373 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:03:46.464283    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e38019f0-946a-402f-978f-b09315e01913-openshift-service-ca\") pod \"maas-api-e2e-ait-9fab72d8-59c4fd9fd8-8gb4m\" (UID: \"e38019f0-946a-402f-978f-b09315e01913\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9fab72d8-59c4fd9fd8-8gb4m"
Jul 23 12:03:46.533170 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:03:46.533136    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:03:46.565659 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:03:46.565580    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rm54g\" (UniqueName: \"kubernetes.io/projected/e38019f0-946a-402f-978f-b09315e01913-kube-api-access-rm54g\") pod \"maas-api-e2e-ait-9fab72d8-59c4fd9fd8-8gb4m\" (UID: \"e38019f0-946a-402f-978f-b09315e01913\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9fab72d8-59c4fd9fd8-8gb4m"
Jul 23 12:03:46.565805 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:03:46.565656    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e38019f0-946a-402f-978f-b09315e01913-maas-api-tls\") pod \"maas-api-e2e-ait-9fab72d8-59c4fd9fd8-8gb4m\" (UID: \"e38019f0-946a-402f-978f-b09315e01913\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9fab72d8-59c4fd9fd8-8gb4m"
Jul 23 12:03:46.565805 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:03:46.565758    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e38019f0-946a-402f-978f-b09315e01913-openshift-service-ca\") pod \"maas-api-e2e-ait-9fab72d8-59c4fd9fd8-8gb4m\" (UID: \"e38019f0-946a-402f-978f-b09315e01913\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9fab72d8-59c4fd9fd8-8gb4m"
Jul 23 12:03:46.565888 ip-10-0-141-229 kubenswrapper[2563]: E0723 12:03:46.565804    2563 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-9fab72d8-serving-cert: secret "maas-api-e2e-ait-9fab72d8-serving-cert" not found
Jul 23 12:03:46.565888 ip-10-0-141-229 kubenswrapper[2563]: E0723 12:03:46.565866    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e38019f0-946a-402f-978f-b09315e01913-maas-api-tls podName:e38019f0-946a-402f-978f-b09315e01913 nodeName:}" failed. No retries permitted until 2026-07-23 12:03:47.065851563 +0000 UTC m=+1681.458789329 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e38019f0-946a-402f-978f-b09315e01913-maas-api-tls") pod "maas-api-e2e-ait-9fab72d8-59c4fd9fd8-8gb4m" (UID: "e38019f0-946a-402f-978f-b09315e01913") : secret "maas-api-e2e-ait-9fab72d8-serving-cert" not found
Jul 23 12:03:46.566396 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:03:46.566376    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e38019f0-946a-402f-978f-b09315e01913-openshift-service-ca\") pod \"maas-api-e2e-ait-9fab72d8-59c4fd9fd8-8gb4m\" (UID: \"e38019f0-946a-402f-978f-b09315e01913\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9fab72d8-59c4fd9fd8-8gb4m"
Jul 23 12:03:46.601886 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:03:46.601856    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rm54g\" (UniqueName: \"kubernetes.io/projected/e38019f0-946a-402f-978f-b09315e01913-kube-api-access-rm54g\") pod \"maas-api-e2e-ait-9fab72d8-59c4fd9fd8-8gb4m\" (UID: \"e38019f0-946a-402f-978f-b09315e01913\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9fab72d8-59c4fd9fd8-8gb4m"
Jul 23 12:03:46.815567 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:03:46.815528    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:03:47.070953 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:03:47.070908    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e38019f0-946a-402f-978f-b09315e01913-maas-api-tls\") pod \"maas-api-e2e-ait-9fab72d8-59c4fd9fd8-8gb4m\" (UID: \"e38019f0-946a-402f-978f-b09315e01913\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9fab72d8-59c4fd9fd8-8gb4m"
Jul 23 12:03:47.073752 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:03:47.073722    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e38019f0-946a-402f-978f-b09315e01913-maas-api-tls\") pod \"maas-api-e2e-ait-9fab72d8-59c4fd9fd8-8gb4m\" (UID: \"e38019f0-946a-402f-978f-b09315e01913\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9fab72d8-59c4fd9fd8-8gb4m"
Jul 23 12:03:47.289188 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:03:47.289155    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:03:47.320632 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:03:47.320591    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9fab72d8-59c4fd9fd8-8gb4m"
Jul 23 12:03:47.390366 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:03:47.390276    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:03:47.488542 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:03:47.488514    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-9fab72d8-59c4fd9fd8-8gb4m"]
Jul 23 12:03:47.491206 ip-10-0-141-229 kubenswrapper[2563]: W0723 12:03:47.491174    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode38019f0_946a_402f_978f_b09315e01913.slice/crio-fe6ed94b4f7cfab6bf5ded34010d8f6d406c3d1af1b1816d9542801558ea0163 WatchSource:0}: Error finding container fe6ed94b4f7cfab6bf5ded34010d8f6d406c3d1af1b1816d9542801558ea0163: Status 404 returned error can't find the container with id fe6ed94b4f7cfab6bf5ded34010d8f6d406c3d1af1b1816d9542801558ea0163
Jul 23 12:03:47.492776 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:03:47.492745    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 23 12:03:47.928249 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:03:47.928211    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9fab72d8-59c4fd9fd8-8gb4m" event={"ID":"e38019f0-946a-402f-978f-b09315e01913","Type":"ContainerStarted","Data":"fe6ed94b4f7cfab6bf5ded34010d8f6d406c3d1af1b1816d9542801558ea0163"}
Jul 23 12:03:48.934178 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:03:48.934141    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9fab72d8-59c4fd9fd8-8gb4m" event={"ID":"e38019f0-946a-402f-978f-b09315e01913","Type":"ContainerStarted","Data":"b00958552d1fa3385653d57ea3f44acc440d4ca5f06a1613943a71731a63485d"}
Jul 23 12:03:48.934597 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:03:48.934268    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9fab72d8-59c4fd9fd8-8gb4m"
Jul 23 12:03:48.959717 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:03:48.959647    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9fab72d8-59c4fd9fd8-8gb4m" podStartSLOduration=2.299872965 podStartE2EDuration="2.959631284s" totalImagesPullingTime="659.758319ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 12:03:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 12:03:48.95740486 +0000 UTC m=+1683.350342647" watchObservedRunningTime="2026-07-23 12:03:48.959631284 +0000 UTC m=+1683.352569069"
Jul 23 12:03:54.945585 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:03:54.945554    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9fab72d8-59c4fd9fd8-8gb4m"
Jul 23 12:04:22.567727 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:04:22.567615    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-9fab72d8-59c4fd9fd8-8gb4m"]
Jul 23 12:04:22.568267 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:04:22.567853    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9fab72d8-59c4fd9fd8-8gb4m" podUID="e38019f0-946a-402f-978f-b09315e01913" containerName="maas-api" containerID="cri-o://b00958552d1fa3385653d57ea3f44acc440d4ca5f06a1613943a71731a63485d" gracePeriod=30
Jul 23 12:04:22.597446 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:04:22.597411    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:04:22.822918 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:04:22.822842    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9fab72d8-59c4fd9fd8-8gb4m"
Jul 23 12:04:22.928247 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:04:22.928206    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rm54g\" (UniqueName: \"kubernetes.io/projected/e38019f0-946a-402f-978f-b09315e01913-kube-api-access-rm54g\") pod \"e38019f0-946a-402f-978f-b09315e01913\" (UID: \"e38019f0-946a-402f-978f-b09315e01913\") "
Jul 23 12:04:22.928247 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:04:22.928246    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e38019f0-946a-402f-978f-b09315e01913-openshift-service-ca\") pod \"e38019f0-946a-402f-978f-b09315e01913\" (UID: \"e38019f0-946a-402f-978f-b09315e01913\") "
Jul 23 12:04:22.928466 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:04:22.928270    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e38019f0-946a-402f-978f-b09315e01913-maas-api-tls\") pod \"e38019f0-946a-402f-978f-b09315e01913\" (UID: \"e38019f0-946a-402f-978f-b09315e01913\") "
Jul 23 12:04:22.928646 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:04:22.928622    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e38019f0-946a-402f-978f-b09315e01913-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "e38019f0-946a-402f-978f-b09315e01913" (UID: "e38019f0-946a-402f-978f-b09315e01913"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 12:04:22.930662 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:04:22.930623    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e38019f0-946a-402f-978f-b09315e01913-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "e38019f0-946a-402f-978f-b09315e01913" (UID: "e38019f0-946a-402f-978f-b09315e01913"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 12:04:22.930808 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:04:22.930672    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e38019f0-946a-402f-978f-b09315e01913-kube-api-access-rm54g" (OuterVolumeSpecName: "kube-api-access-rm54g") pod "e38019f0-946a-402f-978f-b09315e01913" (UID: "e38019f0-946a-402f-978f-b09315e01913"). InnerVolumeSpecName "kube-api-access-rm54g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 12:04:23.029313 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:04:23.029270    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rm54g\" (UniqueName: \"kubernetes.io/projected/e38019f0-946a-402f-978f-b09315e01913-kube-api-access-rm54g\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 23 12:04:23.029313 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:04:23.029305    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e38019f0-946a-402f-978f-b09315e01913-openshift-service-ca\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 23 12:04:23.029313 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:04:23.029315    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e38019f0-946a-402f-978f-b09315e01913-maas-api-tls\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 23 12:04:23.095263 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:04:23.095161    2563 generic.go:356] "Generic (PLEG): container finished" podID="e38019f0-946a-402f-978f-b09315e01913" containerID="b00958552d1fa3385653d57ea3f44acc440d4ca5f06a1613943a71731a63485d" exitCode=0
Jul 23 12:04:23.095263 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:04:23.095214    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9fab72d8-59c4fd9fd8-8gb4m"
Jul 23 12:04:23.095496 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:04:23.095216    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9fab72d8-59c4fd9fd8-8gb4m" event={"ID":"e38019f0-946a-402f-978f-b09315e01913","Type":"ContainerDied","Data":"b00958552d1fa3385653d57ea3f44acc440d4ca5f06a1613943a71731a63485d"}
Jul 23 12:04:23.095496 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:04:23.095338    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9fab72d8-59c4fd9fd8-8gb4m" event={"ID":"e38019f0-946a-402f-978f-b09315e01913","Type":"ContainerDied","Data":"fe6ed94b4f7cfab6bf5ded34010d8f6d406c3d1af1b1816d9542801558ea0163"}
Jul 23 12:04:23.095496 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:04:23.095363    2563 scope.go:117] "RemoveContainer" containerID="b00958552d1fa3385653d57ea3f44acc440d4ca5f06a1613943a71731a63485d"
Jul 23 12:04:23.108013 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:04:23.107991    2563 scope.go:117] "RemoveContainer" containerID="b00958552d1fa3385653d57ea3f44acc440d4ca5f06a1613943a71731a63485d"
Jul 23 12:04:23.108300 ip-10-0-141-229 kubenswrapper[2563]: E0723 12:04:23.108279    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b00958552d1fa3385653d57ea3f44acc440d4ca5f06a1613943a71731a63485d\": container with ID starting with b00958552d1fa3385653d57ea3f44acc440d4ca5f06a1613943a71731a63485d not found: ID does not exist" containerID="b00958552d1fa3385653d57ea3f44acc440d4ca5f06a1613943a71731a63485d"
Jul 23 12:04:23.108382 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:04:23.108311    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b00958552d1fa3385653d57ea3f44acc440d4ca5f06a1613943a71731a63485d"} err="failed to get container status \"b00958552d1fa3385653d57ea3f44acc440d4ca5f06a1613943a71731a63485d\": rpc error: code = NotFound desc = could not find container \"b00958552d1fa3385653d57ea3f44acc440d4ca5f06a1613943a71731a63485d\": container with ID starting with b00958552d1fa3385653d57ea3f44acc440d4ca5f06a1613943a71731a63485d not found: ID does not exist"
Jul 23 12:04:23.122107 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:04:23.122074    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-9fab72d8-59c4fd9fd8-8gb4m"]
Jul 23 12:04:23.125615 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:04:23.125586    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-9fab72d8-59c4fd9fd8-8gb4m"]
Jul 23 12:04:24.077963 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:04:24.077927    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e38019f0-946a-402f-978f-b09315e01913" path="/var/lib/kubelet/pods/e38019f0-946a-402f-978f-b09315e01913/volumes"
Jul 23 12:04:27.073055 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:04:27.072986    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:04:27.296668 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:04:27.296638    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:04:27.547551 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:04:27.547516    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:04:27.988334 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:04:27.988292    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:04:28.128055 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:04:28.128009    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:04:28.456031 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:04:28.455994    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:04:28.842039 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:04:28.841915    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:04:30.710346 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:04:30.710311    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:04:30.863788 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:04:30.863750    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-9e27edaf-856df67fc5-bwfqp"]
Jul 23 12:04:30.864134 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:04:30.864119    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e38019f0-946a-402f-978f-b09315e01913" containerName="maas-api"
Jul 23 12:04:30.864202 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:04:30.864136    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="e38019f0-946a-402f-978f-b09315e01913" containerName="maas-api"
Jul 23 12:04:30.864241 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:04:30.864215    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="e38019f0-946a-402f-978f-b09315e01913" containerName="maas-api"
Jul 23 12:04:30.868639 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:04:30.868619    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9e27edaf-856df67fc5-bwfqp"
Jul 23 12:04:30.870980 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:04:30.870958    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-9e27edaf-serving-cert\""
Jul 23 12:04:30.882672 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:04:30.882648    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-9e27edaf-856df67fc5-bwfqp"]
Jul 23 12:04:30.972718 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:04:30.972600    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:04:31.006931 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:04:31.006889    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/12516e9b-07a1-4407-ba93-ad2e9b8e1883-openshift-service-ca\") pod \"maas-api-e2e-migrate-9e27edaf-856df67fc5-bwfqp\" (UID: \"12516e9b-07a1-4407-ba93-ad2e9b8e1883\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9e27edaf-856df67fc5-bwfqp"
Jul 23 12:04:31.007143 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:04:31.006972    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/12516e9b-07a1-4407-ba93-ad2e9b8e1883-maas-api-tls\") pod \"maas-api-e2e-migrate-9e27edaf-856df67fc5-bwfqp\" (UID: \"12516e9b-07a1-4407-ba93-ad2e9b8e1883\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9e27edaf-856df67fc5-bwfqp"
Jul 23 12:04:31.007143 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:04:31.007063    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fft8p\" (UniqueName: \"kubernetes.io/projected/12516e9b-07a1-4407-ba93-ad2e9b8e1883-kube-api-access-fft8p\") pod \"maas-api-e2e-migrate-9e27edaf-856df67fc5-bwfqp\" (UID: \"12516e9b-07a1-4407-ba93-ad2e9b8e1883\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9e27edaf-856df67fc5-bwfqp"
Jul 23 12:04:31.107630 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:04:31.107594    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/12516e9b-07a1-4407-ba93-ad2e9b8e1883-openshift-service-ca\") pod \"maas-api-e2e-migrate-9e27edaf-856df67fc5-bwfqp\" (UID: \"12516e9b-07a1-4407-ba93-ad2e9b8e1883\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9e27edaf-856df67fc5-bwfqp"
Jul 23 12:04:31.107879 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:04:31.107674    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/12516e9b-07a1-4407-ba93-ad2e9b8e1883-maas-api-tls\") pod \"maas-api-e2e-migrate-9e27edaf-856df67fc5-bwfqp\" (UID: \"12516e9b-07a1-4407-ba93-ad2e9b8e1883\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9e27edaf-856df67fc5-bwfqp"
Jul 23 12:04:31.107879 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:04:31.107756    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fft8p\" (UniqueName: \"kubernetes.io/projected/12516e9b-07a1-4407-ba93-ad2e9b8e1883-kube-api-access-fft8p\") pod \"maas-api-e2e-migrate-9e27edaf-856df67fc5-bwfqp\" (UID: \"12516e9b-07a1-4407-ba93-ad2e9b8e1883\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9e27edaf-856df67fc5-bwfqp"
Jul 23 12:04:31.107879 ip-10-0-141-229 kubenswrapper[2563]: E0723 12:04:31.107820    2563 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-9e27edaf-serving-cert: secret "maas-api-e2e-migrate-9e27edaf-serving-cert" not found
Jul 23 12:04:31.108056 ip-10-0-141-229 kubenswrapper[2563]: E0723 12:04:31.107917    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/12516e9b-07a1-4407-ba93-ad2e9b8e1883-maas-api-tls podName:12516e9b-07a1-4407-ba93-ad2e9b8e1883 nodeName:}" failed. No retries permitted until 2026-07-23 12:04:31.607893986 +0000 UTC m=+1726.000831754 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/12516e9b-07a1-4407-ba93-ad2e9b8e1883-maas-api-tls") pod "maas-api-e2e-migrate-9e27edaf-856df67fc5-bwfqp" (UID: "12516e9b-07a1-4407-ba93-ad2e9b8e1883") : secret "maas-api-e2e-migrate-9e27edaf-serving-cert" not found
Jul 23 12:04:31.108336 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:04:31.108316    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/12516e9b-07a1-4407-ba93-ad2e9b8e1883-openshift-service-ca\") pod \"maas-api-e2e-migrate-9e27edaf-856df67fc5-bwfqp\" (UID: \"12516e9b-07a1-4407-ba93-ad2e9b8e1883\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9e27edaf-856df67fc5-bwfqp"
Jul 23 12:04:31.126281 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:04:31.126248    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fft8p\" (UniqueName: \"kubernetes.io/projected/12516e9b-07a1-4407-ba93-ad2e9b8e1883-kube-api-access-fft8p\") pod \"maas-api-e2e-migrate-9e27edaf-856df67fc5-bwfqp\" (UID: \"12516e9b-07a1-4407-ba93-ad2e9b8e1883\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9e27edaf-856df67fc5-bwfqp"
Jul 23 12:04:31.340501 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:04:31.340410    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:04:31.540552 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:04:31.540520    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:04:31.613314 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:04:31.613278    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/12516e9b-07a1-4407-ba93-ad2e9b8e1883-maas-api-tls\") pod \"maas-api-e2e-migrate-9e27edaf-856df67fc5-bwfqp\" (UID: \"12516e9b-07a1-4407-ba93-ad2e9b8e1883\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9e27edaf-856df67fc5-bwfqp"
Jul 23 12:04:31.616325 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:04:31.616279    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/12516e9b-07a1-4407-ba93-ad2e9b8e1883-maas-api-tls\") pod \"maas-api-e2e-migrate-9e27edaf-856df67fc5-bwfqp\" (UID: \"12516e9b-07a1-4407-ba93-ad2e9b8e1883\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9e27edaf-856df67fc5-bwfqp"
Jul 23 12:04:31.649977 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:04:31.649925    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:04:31.779283 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:04:31.779239    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9e27edaf-856df67fc5-bwfqp"
Jul 23 12:04:31.935418 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:04:31.934575    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-9e27edaf-856df67fc5-bwfqp"]
Jul 23 12:04:32.140245 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:04:32.140149    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9e27edaf-856df67fc5-bwfqp" event={"ID":"12516e9b-07a1-4407-ba93-ad2e9b8e1883","Type":"ContainerStarted","Data":"38f8e3ffd9936c01ca609b2accbd7585aca38fe235bec332d4ff8efd349c1a05"}
Jul 23 12:04:33.146533 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:04:33.146493    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9e27edaf-856df67fc5-bwfqp" event={"ID":"12516e9b-07a1-4407-ba93-ad2e9b8e1883","Type":"ContainerStarted","Data":"69c8a0ef56c9e41bcefd04dc3d3e8f1ad970df090358777b1aab25ac98680103"}
Jul 23 12:04:33.147008 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:04:33.146566    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9e27edaf-856df67fc5-bwfqp"
Jul 23 12:04:33.184930 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:04:33.184858    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9e27edaf-856df67fc5-bwfqp" podStartSLOduration=2.624490602 podStartE2EDuration="3.18483573s" totalImagesPullingTime="560.345128ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 12:04:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 12:04:33.181800569 +0000 UTC m=+1727.574738360" watchObservedRunningTime="2026-07-23 12:04:33.18483573 +0000 UTC m=+1727.577773519"
Jul 23 12:04:39.159408 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:04:39.159380    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9e27edaf-856df67fc5-bwfqp"
Jul 23 12:05:06.309263 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:05:06.309225    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-9e27edaf-856df67fc5-bwfqp"]
Jul 23 12:05:06.309850 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:05:06.309483    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9e27edaf-856df67fc5-bwfqp" podUID="12516e9b-07a1-4407-ba93-ad2e9b8e1883" containerName="maas-api" containerID="cri-o://69c8a0ef56c9e41bcefd04dc3d3e8f1ad970df090358777b1aab25ac98680103" gracePeriod=30
Jul 23 12:05:06.354817 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:05:06.354788    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:05:06.562751 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:05:06.562670    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9e27edaf-856df67fc5-bwfqp"
Jul 23 12:05:06.645778 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:05:06.645738    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fft8p\" (UniqueName: \"kubernetes.io/projected/12516e9b-07a1-4407-ba93-ad2e9b8e1883-kube-api-access-fft8p\") pod \"12516e9b-07a1-4407-ba93-ad2e9b8e1883\" (UID: \"12516e9b-07a1-4407-ba93-ad2e9b8e1883\") "
Jul 23 12:05:06.645778 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:05:06.645784    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/12516e9b-07a1-4407-ba93-ad2e9b8e1883-maas-api-tls\") pod \"12516e9b-07a1-4407-ba93-ad2e9b8e1883\" (UID: \"12516e9b-07a1-4407-ba93-ad2e9b8e1883\") "
Jul 23 12:05:06.646002 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:05:06.645853    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/12516e9b-07a1-4407-ba93-ad2e9b8e1883-openshift-service-ca\") pod \"12516e9b-07a1-4407-ba93-ad2e9b8e1883\" (UID: \"12516e9b-07a1-4407-ba93-ad2e9b8e1883\") "
Jul 23 12:05:06.646240 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:05:06.646216    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12516e9b-07a1-4407-ba93-ad2e9b8e1883-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "12516e9b-07a1-4407-ba93-ad2e9b8e1883" (UID: "12516e9b-07a1-4407-ba93-ad2e9b8e1883"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 12:05:06.648204 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:05:06.648175    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12516e9b-07a1-4407-ba93-ad2e9b8e1883-kube-api-access-fft8p" (OuterVolumeSpecName: "kube-api-access-fft8p") pod "12516e9b-07a1-4407-ba93-ad2e9b8e1883" (UID: "12516e9b-07a1-4407-ba93-ad2e9b8e1883"). InnerVolumeSpecName "kube-api-access-fft8p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 12:05:06.648204 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:05:06.648200    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/12516e9b-07a1-4407-ba93-ad2e9b8e1883-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "12516e9b-07a1-4407-ba93-ad2e9b8e1883" (UID: "12516e9b-07a1-4407-ba93-ad2e9b8e1883"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 12:05:06.747313 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:05:06.747271    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fft8p\" (UniqueName: \"kubernetes.io/projected/12516e9b-07a1-4407-ba93-ad2e9b8e1883-kube-api-access-fft8p\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 23 12:05:06.747313 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:05:06.747307    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/12516e9b-07a1-4407-ba93-ad2e9b8e1883-maas-api-tls\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 23 12:05:06.747313 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:05:06.747316    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/12516e9b-07a1-4407-ba93-ad2e9b8e1883-openshift-service-ca\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 23 12:05:07.314745 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:05:07.314667    2563 generic.go:356] "Generic (PLEG): container finished" podID="12516e9b-07a1-4407-ba93-ad2e9b8e1883" containerID="69c8a0ef56c9e41bcefd04dc3d3e8f1ad970df090358777b1aab25ac98680103" exitCode=0
Jul 23 12:05:07.314745 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:05:07.314730    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9e27edaf-856df67fc5-bwfqp"
Jul 23 12:05:07.315242 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:05:07.314761    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9e27edaf-856df67fc5-bwfqp" event={"ID":"12516e9b-07a1-4407-ba93-ad2e9b8e1883","Type":"ContainerDied","Data":"69c8a0ef56c9e41bcefd04dc3d3e8f1ad970df090358777b1aab25ac98680103"}
Jul 23 12:05:07.315242 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:05:07.314819    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9e27edaf-856df67fc5-bwfqp" event={"ID":"12516e9b-07a1-4407-ba93-ad2e9b8e1883","Type":"ContainerDied","Data":"38f8e3ffd9936c01ca609b2accbd7585aca38fe235bec332d4ff8efd349c1a05"}
Jul 23 12:05:07.315242 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:05:07.314844    2563 scope.go:117] "RemoveContainer" containerID="69c8a0ef56c9e41bcefd04dc3d3e8f1ad970df090358777b1aab25ac98680103"
Jul 23 12:05:07.330753 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:05:07.330731    2563 scope.go:117] "RemoveContainer" containerID="69c8a0ef56c9e41bcefd04dc3d3e8f1ad970df090358777b1aab25ac98680103"
Jul 23 12:05:07.331054 ip-10-0-141-229 kubenswrapper[2563]: E0723 12:05:07.331025    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69c8a0ef56c9e41bcefd04dc3d3e8f1ad970df090358777b1aab25ac98680103\": container with ID starting with 69c8a0ef56c9e41bcefd04dc3d3e8f1ad970df090358777b1aab25ac98680103 not found: ID does not exist" containerID="69c8a0ef56c9e41bcefd04dc3d3e8f1ad970df090358777b1aab25ac98680103"
Jul 23 12:05:07.331113 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:05:07.331069    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69c8a0ef56c9e41bcefd04dc3d3e8f1ad970df090358777b1aab25ac98680103"} err="failed to get container status \"69c8a0ef56c9e41bcefd04dc3d3e8f1ad970df090358777b1aab25ac98680103\": rpc error: code = NotFound desc = could not find container \"69c8a0ef56c9e41bcefd04dc3d3e8f1ad970df090358777b1aab25ac98680103\": container with ID starting with 69c8a0ef56c9e41bcefd04dc3d3e8f1ad970df090358777b1aab25ac98680103 not found: ID does not exist"
Jul 23 12:05:07.344186 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:05:07.344156    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-9e27edaf-856df67fc5-bwfqp"]
Jul 23 12:05:07.347850 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:05:07.347823    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-9e27edaf-856df67fc5-bwfqp"]
Jul 23 12:05:08.077938 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:05:08.077901    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="12516e9b-07a1-4407-ba93-ad2e9b8e1883" path="/var/lib/kubelet/pods/12516e9b-07a1-4407-ba93-ad2e9b8e1883/volumes"
Jul 23 12:05:10.844828 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:05:10.844787    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:05:11.014604 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:05:11.014544    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:05:11.307086 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:05:11.307050    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:05:11.557210 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:05:11.557118    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:05:11.933610 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:05:11.933575    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:05:12.278982 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:05:12.278878    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:05:14.789483 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:05:14.789448    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:05:14.879011 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:05:14.878974    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c1664359-755c6b4c46-7qhtj"]
Jul 23 12:05:14.879378 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:05:14.879359    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="12516e9b-07a1-4407-ba93-ad2e9b8e1883" containerName="maas-api"
Jul 23 12:05:14.879378 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:05:14.879378    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="12516e9b-07a1-4407-ba93-ad2e9b8e1883" containerName="maas-api"
Jul 23 12:05:14.879482 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:05:14.879455    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="12516e9b-07a1-4407-ba93-ad2e9b8e1883" containerName="maas-api"
Jul 23 12:05:14.883883 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:05:14.883861    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c1664359-755c6b4c46-7qhtj"
Jul 23 12:05:14.888152 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:05:14.888129    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-c1664359-serving-cert\""
Jul 23 12:05:14.897400 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:05:14.897372    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c1664359-755c6b4c46-7qhtj"]
Jul 23 12:05:14.908831 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:05:14.908804    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:05:14.924440 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:05:14.924404    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4d605b18-a557-44e0-a11d-29ee6574fa21-maas-api-tls\") pod \"maas-api-e2e-ait-c1664359-755c6b4c46-7qhtj\" (UID: \"4d605b18-a557-44e0-a11d-29ee6574fa21\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c1664359-755c6b4c46-7qhtj"
Jul 23 12:05:14.924612 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:05:14.924481    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d605b18-a557-44e0-a11d-29ee6574fa21-openshift-service-ca\") pod \"maas-api-e2e-ait-c1664359-755c6b4c46-7qhtj\" (UID: \"4d605b18-a557-44e0-a11d-29ee6574fa21\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c1664359-755c6b4c46-7qhtj"
Jul 23 12:05:14.924679 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:05:14.924643    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7v5b9\" (UniqueName: \"kubernetes.io/projected/4d605b18-a557-44e0-a11d-29ee6574fa21-kube-api-access-7v5b9\") pod \"maas-api-e2e-ait-c1664359-755c6b4c46-7qhtj\" (UID: \"4d605b18-a557-44e0-a11d-29ee6574fa21\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c1664359-755c6b4c46-7qhtj"
Jul 23 12:05:15.012954 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:05:15.012916    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:05:15.025192 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:05:15.025162    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d605b18-a557-44e0-a11d-29ee6574fa21-openshift-service-ca\") pod \"maas-api-e2e-ait-c1664359-755c6b4c46-7qhtj\" (UID: \"4d605b18-a557-44e0-a11d-29ee6574fa21\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c1664359-755c6b4c46-7qhtj"
Jul 23 12:05:15.025315 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:05:15.025236    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7v5b9\" (UniqueName: \"kubernetes.io/projected/4d605b18-a557-44e0-a11d-29ee6574fa21-kube-api-access-7v5b9\") pod \"maas-api-e2e-ait-c1664359-755c6b4c46-7qhtj\" (UID: \"4d605b18-a557-44e0-a11d-29ee6574fa21\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c1664359-755c6b4c46-7qhtj"
Jul 23 12:05:15.025315 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:05:15.025268    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4d605b18-a557-44e0-a11d-29ee6574fa21-maas-api-tls\") pod \"maas-api-e2e-ait-c1664359-755c6b4c46-7qhtj\" (UID: \"4d605b18-a557-44e0-a11d-29ee6574fa21\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c1664359-755c6b4c46-7qhtj"
Jul 23 12:05:15.025410 ip-10-0-141-229 kubenswrapper[2563]: E0723 12:05:15.025365    2563 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-c1664359-serving-cert: secret "maas-api-e2e-ait-c1664359-serving-cert" not found
Jul 23 12:05:15.025451 ip-10-0-141-229 kubenswrapper[2563]: E0723 12:05:15.025429    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4d605b18-a557-44e0-a11d-29ee6574fa21-maas-api-tls podName:4d605b18-a557-44e0-a11d-29ee6574fa21 nodeName:}" failed. No retries permitted until 2026-07-23 12:05:15.525410649 +0000 UTC m=+1769.918348416 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4d605b18-a557-44e0-a11d-29ee6574fa21-maas-api-tls") pod "maas-api-e2e-ait-c1664359-755c6b4c46-7qhtj" (UID: "4d605b18-a557-44e0-a11d-29ee6574fa21") : secret "maas-api-e2e-ait-c1664359-serving-cert" not found
Jul 23 12:05:15.025976 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:05:15.025956    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d605b18-a557-44e0-a11d-29ee6574fa21-openshift-service-ca\") pod \"maas-api-e2e-ait-c1664359-755c6b4c46-7qhtj\" (UID: \"4d605b18-a557-44e0-a11d-29ee6574fa21\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c1664359-755c6b4c46-7qhtj"
Jul 23 12:05:15.037864 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:05:15.037833    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7v5b9\" (UniqueName: \"kubernetes.io/projected/4d605b18-a557-44e0-a11d-29ee6574fa21-kube-api-access-7v5b9\") pod \"maas-api-e2e-ait-c1664359-755c6b4c46-7qhtj\" (UID: \"4d605b18-a557-44e0-a11d-29ee6574fa21\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c1664359-755c6b4c46-7qhtj"
Jul 23 12:05:15.482389 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:05:15.482358    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:05:15.530584 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:05:15.530545    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4d605b18-a557-44e0-a11d-29ee6574fa21-maas-api-tls\") pod \"maas-api-e2e-ait-c1664359-755c6b4c46-7qhtj\" (UID: \"4d605b18-a557-44e0-a11d-29ee6574fa21\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c1664359-755c6b4c46-7qhtj"
Jul 23 12:05:15.530775 ip-10-0-141-229 kubenswrapper[2563]: E0723 12:05:15.530711    2563 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-c1664359-serving-cert: secret "maas-api-e2e-ait-c1664359-serving-cert" not found
Jul 23 12:05:15.530823 ip-10-0-141-229 kubenswrapper[2563]: E0723 12:05:15.530784    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4d605b18-a557-44e0-a11d-29ee6574fa21-maas-api-tls podName:4d605b18-a557-44e0-a11d-29ee6574fa21 nodeName:}" failed. No retries permitted until 2026-07-23 12:05:16.530768602 +0000 UTC m=+1770.923706368 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4d605b18-a557-44e0-a11d-29ee6574fa21-maas-api-tls") pod "maas-api-e2e-ait-c1664359-755c6b4c46-7qhtj" (UID: "4d605b18-a557-44e0-a11d-29ee6574fa21") : secret "maas-api-e2e-ait-c1664359-serving-cert" not found
Jul 23 12:05:15.597891 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:05:15.597855    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:05:16.083161 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:05:16.083102    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:05:16.537163 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:05:16.537127    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4d605b18-a557-44e0-a11d-29ee6574fa21-maas-api-tls\") pod \"maas-api-e2e-ait-c1664359-755c6b4c46-7qhtj\" (UID: \"4d605b18-a557-44e0-a11d-29ee6574fa21\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c1664359-755c6b4c46-7qhtj"
Jul 23 12:05:16.539723 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:05:16.539672    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4d605b18-a557-44e0-a11d-29ee6574fa21-maas-api-tls\") pod \"maas-api-e2e-ait-c1664359-755c6b4c46-7qhtj\" (UID: \"4d605b18-a557-44e0-a11d-29ee6574fa21\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c1664359-755c6b4c46-7qhtj"
Jul 23 12:05:16.694764 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:05:16.694730    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c1664359-755c6b4c46-7qhtj"
Jul 23 12:05:16.834293 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:05:16.834228    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c1664359-755c6b4c46-7qhtj"]
Jul 23 12:05:16.839635 ip-10-0-141-229 kubenswrapper[2563]: W0723 12:05:16.839594    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d605b18_a557_44e0_a11d_29ee6574fa21.slice/crio-4829a98e0fcef71cf6fc93eab40919dce275a3323c741ee0dc0262d4c0711a4c WatchSource:0}: Error finding container 4829a98e0fcef71cf6fc93eab40919dce275a3323c741ee0dc0262d4c0711a4c: Status 404 returned error can't find the container with id 4829a98e0fcef71cf6fc93eab40919dce275a3323c741ee0dc0262d4c0711a4c
Jul 23 12:05:17.373930 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:05:17.373892    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c1664359-755c6b4c46-7qhtj" event={"ID":"4d605b18-a557-44e0-a11d-29ee6574fa21","Type":"ContainerStarted","Data":"4c3feda3b9b18fc844069438ff2b10881853c9b10636ba01062afd5b7776c2fa"}
Jul 23 12:05:17.374362 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:05:17.373956    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c1664359-755c6b4c46-7qhtj" event={"ID":"4d605b18-a557-44e0-a11d-29ee6574fa21","Type":"ContainerStarted","Data":"4829a98e0fcef71cf6fc93eab40919dce275a3323c741ee0dc0262d4c0711a4c"}
Jul 23 12:05:17.374362 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:05:17.374066    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c1664359-755c6b4c46-7qhtj"
Jul 23 12:05:17.405247 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:05:17.405190    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c1664359-755c6b4c46-7qhtj" podStartSLOduration=2.976829996 podStartE2EDuration="3.405171989s" totalImagesPullingTime="428.341993ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 12:05:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 12:05:17.399325422 +0000 UTC m=+1771.792263209" watchObservedRunningTime="2026-07-23 12:05:17.405171989 +0000 UTC m=+1771.798109777"
Jul 23 12:05:23.385346 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:05:23.385314    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c1664359-755c6b4c46-7qhtj"
Jul 23 12:05:46.115995 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:05:46.115954    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nhm9p_93649ac9-9062-46c0-adaa-1048d471211a/ovn-acl-logging/0.log"
Jul 23 12:05:46.120982 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:05:46.120949    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nhm9p_93649ac9-9062-46c0-adaa-1048d471211a/ovn-acl-logging/0.log"
Jul 23 12:05:50.846213 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:05:50.846151    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:05:50.947027 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:05:50.946997    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:05:51.100034 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:05:51.099934    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c1664359-755c6b4c46-7qhtj"]
Jul 23 12:05:51.100220 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:05:51.100191    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c1664359-755c6b4c46-7qhtj" podUID="4d605b18-a557-44e0-a11d-29ee6574fa21" containerName="maas-api" containerID="cri-o://4c3feda3b9b18fc844069438ff2b10881853c9b10636ba01062afd5b7776c2fa" gracePeriod=30
Jul 23 12:05:51.143430 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:05:51.143383    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:05:51.349158 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:05:51.349130    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c1664359-755c6b4c46-7qhtj"
Jul 23 12:05:51.443235 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:05:51.443191    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4d605b18-a557-44e0-a11d-29ee6574fa21-maas-api-tls\") pod \"4d605b18-a557-44e0-a11d-29ee6574fa21\" (UID: \"4d605b18-a557-44e0-a11d-29ee6574fa21\") "
Jul 23 12:05:51.443429 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:05:51.443284    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d605b18-a557-44e0-a11d-29ee6574fa21-openshift-service-ca\") pod \"4d605b18-a557-44e0-a11d-29ee6574fa21\" (UID: \"4d605b18-a557-44e0-a11d-29ee6574fa21\") "
Jul 23 12:05:51.443429 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:05:51.443325    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7v5b9\" (UniqueName: \"kubernetes.io/projected/4d605b18-a557-44e0-a11d-29ee6574fa21-kube-api-access-7v5b9\") pod \"4d605b18-a557-44e0-a11d-29ee6574fa21\" (UID: \"4d605b18-a557-44e0-a11d-29ee6574fa21\") "
Jul 23 12:05:51.443710 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:05:51.443653    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d605b18-a557-44e0-a11d-29ee6574fa21-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "4d605b18-a557-44e0-a11d-29ee6574fa21" (UID: "4d605b18-a557-44e0-a11d-29ee6574fa21"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 12:05:51.445578 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:05:51.445554    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d605b18-a557-44e0-a11d-29ee6574fa21-kube-api-access-7v5b9" (OuterVolumeSpecName: "kube-api-access-7v5b9") pod "4d605b18-a557-44e0-a11d-29ee6574fa21" (UID: "4d605b18-a557-44e0-a11d-29ee6574fa21"). InnerVolumeSpecName "kube-api-access-7v5b9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 12:05:51.445651 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:05:51.445589    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d605b18-a557-44e0-a11d-29ee6574fa21-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "4d605b18-a557-44e0-a11d-29ee6574fa21" (UID: "4d605b18-a557-44e0-a11d-29ee6574fa21"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 12:05:51.537871 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:05:51.537830    2563 generic.go:356] "Generic (PLEG): container finished" podID="4d605b18-a557-44e0-a11d-29ee6574fa21" containerID="4c3feda3b9b18fc844069438ff2b10881853c9b10636ba01062afd5b7776c2fa" exitCode=0
Jul 23 12:05:51.537871 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:05:51.537872    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c1664359-755c6b4c46-7qhtj"
Jul 23 12:05:51.538079 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:05:51.537888    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c1664359-755c6b4c46-7qhtj" event={"ID":"4d605b18-a557-44e0-a11d-29ee6574fa21","Type":"ContainerDied","Data":"4c3feda3b9b18fc844069438ff2b10881853c9b10636ba01062afd5b7776c2fa"}
Jul 23 12:05:51.538079 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:05:51.537943    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c1664359-755c6b4c46-7qhtj" event={"ID":"4d605b18-a557-44e0-a11d-29ee6574fa21","Type":"ContainerDied","Data":"4829a98e0fcef71cf6fc93eab40919dce275a3323c741ee0dc0262d4c0711a4c"}
Jul 23 12:05:51.538079 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:05:51.537975    2563 scope.go:117] "RemoveContainer" containerID="4c3feda3b9b18fc844069438ff2b10881853c9b10636ba01062afd5b7776c2fa"
Jul 23 12:05:51.544276 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:05:51.544234    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7v5b9\" (UniqueName: \"kubernetes.io/projected/4d605b18-a557-44e0-a11d-29ee6574fa21-kube-api-access-7v5b9\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 23 12:05:51.544276 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:05:51.544269    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4d605b18-a557-44e0-a11d-29ee6574fa21-maas-api-tls\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 23 12:05:51.544488 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:05:51.544284    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d605b18-a557-44e0-a11d-29ee6574fa21-openshift-service-ca\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 23 12:05:51.549497 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:05:51.549313    2563 scope.go:117] "RemoveContainer" containerID="4c3feda3b9b18fc844069438ff2b10881853c9b10636ba01062afd5b7776c2fa"
Jul 23 12:05:51.550184 ip-10-0-141-229 kubenswrapper[2563]: E0723 12:05:51.550159    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c3feda3b9b18fc844069438ff2b10881853c9b10636ba01062afd5b7776c2fa\": container with ID starting with 4c3feda3b9b18fc844069438ff2b10881853c9b10636ba01062afd5b7776c2fa not found: ID does not exist" containerID="4c3feda3b9b18fc844069438ff2b10881853c9b10636ba01062afd5b7776c2fa"
Jul 23 12:05:51.550284 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:05:51.550195    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c3feda3b9b18fc844069438ff2b10881853c9b10636ba01062afd5b7776c2fa"} err="failed to get container status \"4c3feda3b9b18fc844069438ff2b10881853c9b10636ba01062afd5b7776c2fa\": rpc error: code = NotFound desc = could not find container \"4c3feda3b9b18fc844069438ff2b10881853c9b10636ba01062afd5b7776c2fa\": container with ID starting with 4c3feda3b9b18fc844069438ff2b10881853c9b10636ba01062afd5b7776c2fa not found: ID does not exist"
Jul 23 12:05:51.567672 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:05:51.567639    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c1664359-755c6b4c46-7qhtj"]
Jul 23 12:05:51.575587 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:05:51.575563    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c1664359-755c6b4c46-7qhtj"]
Jul 23 12:05:52.078454 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:05:52.078414    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4d605b18-a557-44e0-a11d-29ee6574fa21" path="/var/lib/kubelet/pods/4d605b18-a557-44e0-a11d-29ee6574fa21/volumes"
Jul 23 12:05:55.900756 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:05:55.900723    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:05:57.186111 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:05:57.186062    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:05:57.409123 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:05:57.409079    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:05:57.652538 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:05:57.652503    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:05:58.017476 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:05:58.017386    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:05:58.375613 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:05:58.375578    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:06:00.110856 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:06:00.110809    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:06:00.211125 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:06:00.211089    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-c2bbf915-b74dc94b8-c8h5b"]
Jul 23 12:06:00.211452 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:06:00.211440    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d605b18-a557-44e0-a11d-29ee6574fa21" containerName="maas-api"
Jul 23 12:06:00.211497 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:06:00.211454    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d605b18-a557-44e0-a11d-29ee6574fa21" containerName="maas-api"
Jul 23 12:06:00.211531 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:06:00.211522    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="4d605b18-a557-44e0-a11d-29ee6574fa21" containerName="maas-api"
Jul 23 12:06:00.215711 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:06:00.215673    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c2bbf915-b74dc94b8-c8h5b"
Jul 23 12:06:00.220214 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:06:00.220191    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-c2bbf915-serving-cert\""
Jul 23 12:06:00.234323 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:06:00.234272    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:06:00.238124 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:06:00.238095    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-c2bbf915-b74dc94b8-c8h5b"]
Jul 23 12:06:00.327169 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:06:00.327130    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w58cd\" (UniqueName: \"kubernetes.io/projected/b25e8a1c-2a32-4067-bcf4-6b67f3bbb60d-kube-api-access-w58cd\") pod \"maas-api-e2e-derive-c2bbf915-b74dc94b8-c8h5b\" (UID: \"b25e8a1c-2a32-4067-bcf4-6b67f3bbb60d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c2bbf915-b74dc94b8-c8h5b"
Jul 23 12:06:00.327169 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:06:00.327171    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b25e8a1c-2a32-4067-bcf4-6b67f3bbb60d-openshift-service-ca\") pod \"maas-api-e2e-derive-c2bbf915-b74dc94b8-c8h5b\" (UID: \"b25e8a1c-2a32-4067-bcf4-6b67f3bbb60d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c2bbf915-b74dc94b8-c8h5b"
Jul 23 12:06:00.327407 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:06:00.327283    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b25e8a1c-2a32-4067-bcf4-6b67f3bbb60d-maas-api-tls\") pod \"maas-api-e2e-derive-c2bbf915-b74dc94b8-c8h5b\" (UID: \"b25e8a1c-2a32-4067-bcf4-6b67f3bbb60d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c2bbf915-b74dc94b8-c8h5b"
Jul 23 12:06:00.386432 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:06:00.386405    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:06:00.427859 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:06:00.427820    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b25e8a1c-2a32-4067-bcf4-6b67f3bbb60d-maas-api-tls\") pod \"maas-api-e2e-derive-c2bbf915-b74dc94b8-c8h5b\" (UID: \"b25e8a1c-2a32-4067-bcf4-6b67f3bbb60d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c2bbf915-b74dc94b8-c8h5b"
Jul 23 12:06:00.428015 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:06:00.427922    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w58cd\" (UniqueName: \"kubernetes.io/projected/b25e8a1c-2a32-4067-bcf4-6b67f3bbb60d-kube-api-access-w58cd\") pod \"maas-api-e2e-derive-c2bbf915-b74dc94b8-c8h5b\" (UID: \"b25e8a1c-2a32-4067-bcf4-6b67f3bbb60d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c2bbf915-b74dc94b8-c8h5b"
Jul 23 12:06:00.428015 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:06:00.427946    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b25e8a1c-2a32-4067-bcf4-6b67f3bbb60d-openshift-service-ca\") pod \"maas-api-e2e-derive-c2bbf915-b74dc94b8-c8h5b\" (UID: \"b25e8a1c-2a32-4067-bcf4-6b67f3bbb60d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c2bbf915-b74dc94b8-c8h5b"
Jul 23 12:06:00.428618 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:06:00.428593    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b25e8a1c-2a32-4067-bcf4-6b67f3bbb60d-openshift-service-ca\") pod \"maas-api-e2e-derive-c2bbf915-b74dc94b8-c8h5b\" (UID: \"b25e8a1c-2a32-4067-bcf4-6b67f3bbb60d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c2bbf915-b74dc94b8-c8h5b"
Jul 23 12:06:00.430377 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:06:00.430349    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b25e8a1c-2a32-4067-bcf4-6b67f3bbb60d-maas-api-tls\") pod \"maas-api-e2e-derive-c2bbf915-b74dc94b8-c8h5b\" (UID: \"b25e8a1c-2a32-4067-bcf4-6b67f3bbb60d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c2bbf915-b74dc94b8-c8h5b"
Jul 23 12:06:00.446492 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:06:00.446470    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w58cd\" (UniqueName: \"kubernetes.io/projected/b25e8a1c-2a32-4067-bcf4-6b67f3bbb60d-kube-api-access-w58cd\") pod \"maas-api-e2e-derive-c2bbf915-b74dc94b8-c8h5b\" (UID: \"b25e8a1c-2a32-4067-bcf4-6b67f3bbb60d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c2bbf915-b74dc94b8-c8h5b"
Jul 23 12:06:00.525888 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:06:00.525857    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c2bbf915-b74dc94b8-c8h5b"
Jul 23 12:06:00.675369 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:06:00.675345    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-c2bbf915-b74dc94b8-c8h5b"]
Jul 23 12:06:00.677544 ip-10-0-141-229 kubenswrapper[2563]: W0723 12:06:00.677516    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb25e8a1c_2a32_4067_bcf4_6b67f3bbb60d.slice/crio-a52749a33bfbf13472c5ca85f9dbdbe61d262d0c4bdbb27c9189cd6fc3841b64 WatchSource:0}: Error finding container a52749a33bfbf13472c5ca85f9dbdbe61d262d0c4bdbb27c9189cd6fc3841b64: Status 404 returned error can't find the container with id a52749a33bfbf13472c5ca85f9dbdbe61d262d0c4bdbb27c9189cd6fc3841b64
Jul 23 12:06:00.800493 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:06:00.800448    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:06:01.128059 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:06:01.128025    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:06:01.258131 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:06:01.258024    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:06:01.589410 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:06:01.589309    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c2bbf915-b74dc94b8-c8h5b" event={"ID":"b25e8a1c-2a32-4067-bcf4-6b67f3bbb60d","Type":"ContainerStarted","Data":"4356fdda1e3d8a22e73b37e7d787a94477df28d8bf01b1e0f42724707518f471"}
Jul 23 12:06:01.589410 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:06:01.589392    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c2bbf915-b74dc94b8-c8h5b" event={"ID":"b25e8a1c-2a32-4067-bcf4-6b67f3bbb60d","Type":"ContainerStarted","Data":"a52749a33bfbf13472c5ca85f9dbdbe61d262d0c4bdbb27c9189cd6fc3841b64"}
Jul 23 12:06:01.589631 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:06:01.589513    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c2bbf915-b74dc94b8-c8h5b"
Jul 23 12:06:01.612920 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:06:01.612855    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c2bbf915-b74dc94b8-c8h5b" podStartSLOduration=1.310332599 podStartE2EDuration="1.612834999s" totalImagesPullingTime="302.5024ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 12:06:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 12:06:01.611148282 +0000 UTC m=+1816.004086069" watchObservedRunningTime="2026-07-23 12:06:01.612834999 +0000 UTC m=+1816.005772789"
Jul 23 12:06:07.600348 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:06:07.600316    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c2bbf915-b74dc94b8-c8h5b"
Jul 23 12:06:29.399248 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:06:29.399214    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:06:35.728349 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:06:35.728317    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-c2bbf915-b74dc94b8-c8h5b"]
Jul 23 12:06:35.728846 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:06:35.728553    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c2bbf915-b74dc94b8-c8h5b" podUID="b25e8a1c-2a32-4067-bcf4-6b67f3bbb60d" containerName="maas-api" containerID="cri-o://4356fdda1e3d8a22e73b37e7d787a94477df28d8bf01b1e0f42724707518f471" gracePeriod=30
Jul 23 12:06:35.768473 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:06:35.768436    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:06:35.976538 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:06:35.976511    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c2bbf915-b74dc94b8-c8h5b"
Jul 23 12:06:36.050761 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:06:36.050645    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b25e8a1c-2a32-4067-bcf4-6b67f3bbb60d-maas-api-tls\") pod \"b25e8a1c-2a32-4067-bcf4-6b67f3bbb60d\" (UID: \"b25e8a1c-2a32-4067-bcf4-6b67f3bbb60d\") "
Jul 23 12:06:36.050761 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:06:36.050720    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w58cd\" (UniqueName: \"kubernetes.io/projected/b25e8a1c-2a32-4067-bcf4-6b67f3bbb60d-kube-api-access-w58cd\") pod \"b25e8a1c-2a32-4067-bcf4-6b67f3bbb60d\" (UID: \"b25e8a1c-2a32-4067-bcf4-6b67f3bbb60d\") "
Jul 23 12:06:36.050947 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:06:36.050810    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b25e8a1c-2a32-4067-bcf4-6b67f3bbb60d-openshift-service-ca\") pod \"b25e8a1c-2a32-4067-bcf4-6b67f3bbb60d\" (UID: \"b25e8a1c-2a32-4067-bcf4-6b67f3bbb60d\") "
Jul 23 12:06:36.051158 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:06:36.051128    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b25e8a1c-2a32-4067-bcf4-6b67f3bbb60d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "b25e8a1c-2a32-4067-bcf4-6b67f3bbb60d" (UID: "b25e8a1c-2a32-4067-bcf4-6b67f3bbb60d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 12:06:36.053006 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:06:36.052971    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b25e8a1c-2a32-4067-bcf4-6b67f3bbb60d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "b25e8a1c-2a32-4067-bcf4-6b67f3bbb60d" (UID: "b25e8a1c-2a32-4067-bcf4-6b67f3bbb60d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 12:06:36.053106 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:06:36.053058    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b25e8a1c-2a32-4067-bcf4-6b67f3bbb60d-kube-api-access-w58cd" (OuterVolumeSpecName: "kube-api-access-w58cd") pod "b25e8a1c-2a32-4067-bcf4-6b67f3bbb60d" (UID: "b25e8a1c-2a32-4067-bcf4-6b67f3bbb60d"). InnerVolumeSpecName "kube-api-access-w58cd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 12:06:36.152491 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:06:36.152454    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b25e8a1c-2a32-4067-bcf4-6b67f3bbb60d-maas-api-tls\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 23 12:06:36.152647 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:06:36.152579    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w58cd\" (UniqueName: \"kubernetes.io/projected/b25e8a1c-2a32-4067-bcf4-6b67f3bbb60d-kube-api-access-w58cd\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 23 12:06:36.152647 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:06:36.152598    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b25e8a1c-2a32-4067-bcf4-6b67f3bbb60d-openshift-service-ca\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 23 12:06:36.766009 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:06:36.765974    2563 generic.go:356] "Generic (PLEG): container finished" podID="b25e8a1c-2a32-4067-bcf4-6b67f3bbb60d" containerID="4356fdda1e3d8a22e73b37e7d787a94477df28d8bf01b1e0f42724707518f471" exitCode=0
Jul 23 12:06:36.766009 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:06:36.766015    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c2bbf915-b74dc94b8-c8h5b"
Jul 23 12:06:36.766537 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:06:36.766053    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c2bbf915-b74dc94b8-c8h5b" event={"ID":"b25e8a1c-2a32-4067-bcf4-6b67f3bbb60d","Type":"ContainerDied","Data":"4356fdda1e3d8a22e73b37e7d787a94477df28d8bf01b1e0f42724707518f471"}
Jul 23 12:06:36.766537 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:06:36.766108    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c2bbf915-b74dc94b8-c8h5b" event={"ID":"b25e8a1c-2a32-4067-bcf4-6b67f3bbb60d","Type":"ContainerDied","Data":"a52749a33bfbf13472c5ca85f9dbdbe61d262d0c4bdbb27c9189cd6fc3841b64"}
Jul 23 12:06:36.766537 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:06:36.766136    2563 scope.go:117] "RemoveContainer" containerID="4356fdda1e3d8a22e73b37e7d787a94477df28d8bf01b1e0f42724707518f471"
Jul 23 12:06:36.774619 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:06:36.774601    2563 scope.go:117] "RemoveContainer" containerID="4356fdda1e3d8a22e73b37e7d787a94477df28d8bf01b1e0f42724707518f471"
Jul 23 12:06:36.774892 ip-10-0-141-229 kubenswrapper[2563]: E0723 12:06:36.774873    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4356fdda1e3d8a22e73b37e7d787a94477df28d8bf01b1e0f42724707518f471\": container with ID starting with 4356fdda1e3d8a22e73b37e7d787a94477df28d8bf01b1e0f42724707518f471 not found: ID does not exist" containerID="4356fdda1e3d8a22e73b37e7d787a94477df28d8bf01b1e0f42724707518f471"
Jul 23 12:06:36.774948 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:06:36.774900    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4356fdda1e3d8a22e73b37e7d787a94477df28d8bf01b1e0f42724707518f471"} err="failed to get container status \"4356fdda1e3d8a22e73b37e7d787a94477df28d8bf01b1e0f42724707518f471\": rpc error: code = NotFound desc = could not find container \"4356fdda1e3d8a22e73b37e7d787a94477df28d8bf01b1e0f42724707518f471\": container with ID starting with 4356fdda1e3d8a22e73b37e7d787a94477df28d8bf01b1e0f42724707518f471 not found: ID does not exist"
Jul 23 12:06:36.788783 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:06:36.788758    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-c2bbf915-b74dc94b8-c8h5b"]
Jul 23 12:06:36.793835 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:06:36.793812    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-c2bbf915-b74dc94b8-c8h5b"]
Jul 23 12:06:38.078613 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:06:38.078566    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b25e8a1c-2a32-4067-bcf4-6b67f3bbb60d" path="/var/lib/kubelet/pods/b25e8a1c-2a32-4067-bcf4-6b67f3bbb60d/volumes"
Jul 23 12:06:40.078993 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:06:40.078950    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:06:40.316500 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:06:40.316454    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:06:40.538029 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:06:40.537992    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:06:41.438864 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:06:41.438829    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:06:41.538543 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:06:41.538490    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:06:42.865916 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:06:42.865867    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:06:49.232495 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:06:49.232456    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:06:49.335636 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:06:49.335603    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:06:49.459919 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:06:49.459886    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:06:49.557211 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:06:49.557126    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:06:50.032766 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:06:50.032730    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:07:16.085563 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:07:16.085480    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:07:23.090720 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:07:23.090663    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:07:23.215134 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:07:23.215100    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:07:23.337708 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:07:23.337645    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:07:30.879110 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:07:30.879061    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:07:36.690625 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:07:36.690587    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:07:39.383794 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:07:39.383761    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:07:45.504635 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:07:45.504598    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:07:45.609067 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:07:45.609035    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:07:45.706589 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:07:45.706556    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:07:56.327243 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:07:56.327196    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:07:57.181155 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:07:57.181129    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:08:06.012577 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:08:06.012543    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:08:06.125765 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:08:06.125729    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:08:06.235249 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:08:06.235215    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:08:06.328428 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:08:06.328339    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:08:07.281263 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:08:07.281233    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:08:07.481482 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:08:07.481437    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:08:09.267896 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:08:09.267842    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:08:11.719336 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:08:11.719305    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:08:15.837374 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:08:15.837345    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:08:17.535091 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:08:17.535054    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:08:23.069845 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:08:23.069808    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:08:23.138601 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:08:23.138563    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:08:23.358086 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:08:23.357988    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:08:23.894488 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:08:23.894454    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:08:24.031624 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:08:24.031579    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:08:26.828183 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:08:26.828147    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:08:26.951338 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:08:26.951301    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:08:27.046992 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:08:27.046959    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-110e698d-b99bcdcb4-krx2h"]
Jul 23 12:08:27.047364 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:08:27.047350    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b25e8a1c-2a32-4067-bcf4-6b67f3bbb60d" containerName="maas-api"
Jul 23 12:08:27.047364 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:08:27.047366    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="b25e8a1c-2a32-4067-bcf4-6b67f3bbb60d" containerName="maas-api"
Jul 23 12:08:27.047472 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:08:27.047431    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="b25e8a1c-2a32-4067-bcf4-6b67f3bbb60d" containerName="maas-api"
Jul 23 12:08:27.050566 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:08:27.050545    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-110e698d-b99bcdcb4-krx2h"
Jul 23 12:08:27.053258 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:08:27.053240    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-110e698d-serving-cert\""
Jul 23 12:08:27.068904 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:08:27.068877    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-110e698d-b99bcdcb4-krx2h"]
Jul 23 12:08:27.103063 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:08:27.102925    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:08:27.168051 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:08:27.168020    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/73d2a882-e250-49b1-9fad-ff09626f22f3-openshift-service-ca\") pod \"maas-api-e2e-mt-110e698d-b99bcdcb4-krx2h\" (UID: \"73d2a882-e250-49b1-9fad-ff09626f22f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-110e698d-b99bcdcb4-krx2h"
Jul 23 12:08:27.168228 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:08:27.168059    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4kwt7\" (UniqueName: \"kubernetes.io/projected/73d2a882-e250-49b1-9fad-ff09626f22f3-kube-api-access-4kwt7\") pod \"maas-api-e2e-mt-110e698d-b99bcdcb4-krx2h\" (UID: \"73d2a882-e250-49b1-9fad-ff09626f22f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-110e698d-b99bcdcb4-krx2h"
Jul 23 12:08:27.168228 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:08:27.168167    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/73d2a882-e250-49b1-9fad-ff09626f22f3-maas-api-tls\") pod \"maas-api-e2e-mt-110e698d-b99bcdcb4-krx2h\" (UID: \"73d2a882-e250-49b1-9fad-ff09626f22f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-110e698d-b99bcdcb4-krx2h"
Jul 23 12:08:27.203515 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:08:27.203468    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:08:27.269383 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:08:27.269344    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/73d2a882-e250-49b1-9fad-ff09626f22f3-maas-api-tls\") pod \"maas-api-e2e-mt-110e698d-b99bcdcb4-krx2h\" (UID: \"73d2a882-e250-49b1-9fad-ff09626f22f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-110e698d-b99bcdcb4-krx2h"
Jul 23 12:08:27.269567 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:08:27.269411    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/73d2a882-e250-49b1-9fad-ff09626f22f3-openshift-service-ca\") pod \"maas-api-e2e-mt-110e698d-b99bcdcb4-krx2h\" (UID: \"73d2a882-e250-49b1-9fad-ff09626f22f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-110e698d-b99bcdcb4-krx2h"
Jul 23 12:08:27.269567 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:08:27.269435    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4kwt7\" (UniqueName: \"kubernetes.io/projected/73d2a882-e250-49b1-9fad-ff09626f22f3-kube-api-access-4kwt7\") pod \"maas-api-e2e-mt-110e698d-b99bcdcb4-krx2h\" (UID: \"73d2a882-e250-49b1-9fad-ff09626f22f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-110e698d-b99bcdcb4-krx2h"
Jul 23 12:08:27.270140 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:08:27.270114    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/73d2a882-e250-49b1-9fad-ff09626f22f3-openshift-service-ca\") pod \"maas-api-e2e-mt-110e698d-b99bcdcb4-krx2h\" (UID: \"73d2a882-e250-49b1-9fad-ff09626f22f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-110e698d-b99bcdcb4-krx2h"
Jul 23 12:08:27.272097 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:08:27.272072    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/73d2a882-e250-49b1-9fad-ff09626f22f3-maas-api-tls\") pod \"maas-api-e2e-mt-110e698d-b99bcdcb4-krx2h\" (UID: \"73d2a882-e250-49b1-9fad-ff09626f22f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-110e698d-b99bcdcb4-krx2h"
Jul 23 12:08:27.281322 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:08:27.281297    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4kwt7\" (UniqueName: \"kubernetes.io/projected/73d2a882-e250-49b1-9fad-ff09626f22f3-kube-api-access-4kwt7\") pod \"maas-api-e2e-mt-110e698d-b99bcdcb4-krx2h\" (UID: \"73d2a882-e250-49b1-9fad-ff09626f22f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-110e698d-b99bcdcb4-krx2h"
Jul 23 12:08:27.360026 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:08:27.359941    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-110e698d-b99bcdcb4-krx2h"
Jul 23 12:08:27.519831 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:08:27.519800    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-110e698d-b99bcdcb4-krx2h"]
Jul 23 12:08:27.521464 ip-10-0-141-229 kubenswrapper[2563]: W0723 12:08:27.521432    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73d2a882_e250_49b1_9fad_ff09626f22f3.slice/crio-b19f7c6fd0f6ba6b03df7dcd48f4b23df228f74d3f5fbaf612852b3c97a35ac6 WatchSource:0}: Error finding container b19f7c6fd0f6ba6b03df7dcd48f4b23df228f74d3f5fbaf612852b3c97a35ac6: Status 404 returned error can't find the container with id b19f7c6fd0f6ba6b03df7dcd48f4b23df228f74d3f5fbaf612852b3c97a35ac6
Jul 23 12:08:27.797449 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:08:27.797417    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:08:27.890815 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:08:27.890784    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:08:28.298186 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:08:28.298142    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-110e698d-b99bcdcb4-krx2h" event={"ID":"73d2a882-e250-49b1-9fad-ff09626f22f3","Type":"ContainerStarted","Data":"74bbd659f4766beca900bb789f0af89753db5611d9f471a63f5264ee5b21ac97"}
Jul 23 12:08:28.298366 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:08:28.298205    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-110e698d-b99bcdcb4-krx2h" event={"ID":"73d2a882-e250-49b1-9fad-ff09626f22f3","Type":"ContainerStarted","Data":"b19f7c6fd0f6ba6b03df7dcd48f4b23df228f74d3f5fbaf612852b3c97a35ac6"}
Jul 23 12:08:28.298366 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:08:28.298347    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-110e698d-b99bcdcb4-krx2h"
Jul 23 12:08:28.340040 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:08:28.339939    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-110e698d-b99bcdcb4-krx2h" podStartSLOduration=0.7740393 podStartE2EDuration="1.339921109s" totalImagesPullingTime="565.881809ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 12:08:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 12:08:28.338523912 +0000 UTC m=+1962.731461700" watchObservedRunningTime="2026-07-23 12:08:28.339921109 +0000 UTC m=+1962.732858897"
Jul 23 12:08:31.304296 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:08:31.304263    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:08:32.047507 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:08:32.047466    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:08:32.136346 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:08:32.136318    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:08:32.518496 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:08:32.518463    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:08:34.309050 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:08:34.309022    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-110e698d-b99bcdcb4-krx2h"
Jul 23 12:08:38.894065 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:08:38.893976    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:08:38.994574 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:08:38.994540    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:08:49.623663 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:08:49.623626    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-110e698d-b99bcdcb4-krx2h"]
Jul 23 12:08:49.624162 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:08:49.623933    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-110e698d-b99bcdcb4-krx2h" podUID="73d2a882-e250-49b1-9fad-ff09626f22f3" containerName="maas-api" containerID="cri-o://74bbd659f4766beca900bb789f0af89753db5611d9f471a63f5264ee5b21ac97" gracePeriod=30
Jul 23 12:08:49.660599 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:08:49.660561    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:08:49.861341 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:08:49.861309    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-110e698d-b99bcdcb4-krx2h"
Jul 23 12:08:49.986918 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:08:49.986830    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/73d2a882-e250-49b1-9fad-ff09626f22f3-openshift-service-ca\") pod \"73d2a882-e250-49b1-9fad-ff09626f22f3\" (UID: \"73d2a882-e250-49b1-9fad-ff09626f22f3\") "
Jul 23 12:08:49.987074 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:08:49.986945    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4kwt7\" (UniqueName: \"kubernetes.io/projected/73d2a882-e250-49b1-9fad-ff09626f22f3-kube-api-access-4kwt7\") pod \"73d2a882-e250-49b1-9fad-ff09626f22f3\" (UID: \"73d2a882-e250-49b1-9fad-ff09626f22f3\") "
Jul 23 12:08:49.987074 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:08:49.986973    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/73d2a882-e250-49b1-9fad-ff09626f22f3-maas-api-tls\") pod \"73d2a882-e250-49b1-9fad-ff09626f22f3\" (UID: \"73d2a882-e250-49b1-9fad-ff09626f22f3\") "
Jul 23 12:08:49.987276 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:08:49.987249    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73d2a882-e250-49b1-9fad-ff09626f22f3-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "73d2a882-e250-49b1-9fad-ff09626f22f3" (UID: "73d2a882-e250-49b1-9fad-ff09626f22f3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 12:08:49.989176 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:08:49.989141    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73d2a882-e250-49b1-9fad-ff09626f22f3-kube-api-access-4kwt7" (OuterVolumeSpecName: "kube-api-access-4kwt7") pod "73d2a882-e250-49b1-9fad-ff09626f22f3" (UID: "73d2a882-e250-49b1-9fad-ff09626f22f3"). InnerVolumeSpecName "kube-api-access-4kwt7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 12:08:49.989282 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:08:49.989243    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/73d2a882-e250-49b1-9fad-ff09626f22f3-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "73d2a882-e250-49b1-9fad-ff09626f22f3" (UID: "73d2a882-e250-49b1-9fad-ff09626f22f3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 12:08:50.088447 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:08:50.088423    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4kwt7\" (UniqueName: \"kubernetes.io/projected/73d2a882-e250-49b1-9fad-ff09626f22f3-kube-api-access-4kwt7\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 23 12:08:50.088447 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:08:50.088449    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/73d2a882-e250-49b1-9fad-ff09626f22f3-maas-api-tls\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 23 12:08:50.088585 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:08:50.088459    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/73d2a882-e250-49b1-9fad-ff09626f22f3-openshift-service-ca\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 23 12:08:50.399504 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:08:50.399469    2563 generic.go:356] "Generic (PLEG): container finished" podID="73d2a882-e250-49b1-9fad-ff09626f22f3" containerID="74bbd659f4766beca900bb789f0af89753db5611d9f471a63f5264ee5b21ac97" exitCode=0
Jul 23 12:08:50.399678 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:08:50.399519    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-110e698d-b99bcdcb4-krx2h"
Jul 23 12:08:50.399678 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:08:50.399558    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-110e698d-b99bcdcb4-krx2h" event={"ID":"73d2a882-e250-49b1-9fad-ff09626f22f3","Type":"ContainerDied","Data":"74bbd659f4766beca900bb789f0af89753db5611d9f471a63f5264ee5b21ac97"}
Jul 23 12:08:50.399678 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:08:50.399614    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-110e698d-b99bcdcb4-krx2h" event={"ID":"73d2a882-e250-49b1-9fad-ff09626f22f3","Type":"ContainerDied","Data":"b19f7c6fd0f6ba6b03df7dcd48f4b23df228f74d3f5fbaf612852b3c97a35ac6"}
Jul 23 12:08:50.399678 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:08:50.399637    2563 scope.go:117] "RemoveContainer" containerID="74bbd659f4766beca900bb789f0af89753db5611d9f471a63f5264ee5b21ac97"
Jul 23 12:08:50.408374 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:08:50.408353    2563 scope.go:117] "RemoveContainer" containerID="74bbd659f4766beca900bb789f0af89753db5611d9f471a63f5264ee5b21ac97"
Jul 23 12:08:50.408662 ip-10-0-141-229 kubenswrapper[2563]: E0723 12:08:50.408642    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74bbd659f4766beca900bb789f0af89753db5611d9f471a63f5264ee5b21ac97\": container with ID starting with 74bbd659f4766beca900bb789f0af89753db5611d9f471a63f5264ee5b21ac97 not found: ID does not exist" containerID="74bbd659f4766beca900bb789f0af89753db5611d9f471a63f5264ee5b21ac97"
Jul 23 12:08:50.408765 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:08:50.408671    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74bbd659f4766beca900bb789f0af89753db5611d9f471a63f5264ee5b21ac97"} err="failed to get container status \"74bbd659f4766beca900bb789f0af89753db5611d9f471a63f5264ee5b21ac97\": rpc error: code = NotFound desc = could not find container \"74bbd659f4766beca900bb789f0af89753db5611d9f471a63f5264ee5b21ac97\": container with ID starting with 74bbd659f4766beca900bb789f0af89753db5611d9f471a63f5264ee5b21ac97 not found: ID does not exist"
Jul 23 12:08:50.437822 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:08:50.437784    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-110e698d-b99bcdcb4-krx2h"]
Jul 23 12:08:50.443138 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:08:50.443113    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-110e698d-b99bcdcb4-krx2h"]
Jul 23 12:08:52.078276 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:08:52.078226    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="73d2a882-e250-49b1-9fad-ff09626f22f3" path="/var/lib/kubelet/pods/73d2a882-e250-49b1-9fad-ff09626f22f3/volumes"
Jul 23 12:08:54.390066 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:08:54.390028    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:08:54.855267 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:08:54.855177    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:09:00.914069 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:09:00.914031    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:09:01.023425 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:09:01.023391    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:09:01.244472 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:09:01.244358    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:09:01.498276 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:09:01.498182    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:09:01.656278 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:09:01.656247    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:09:01.780081 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:09:01.779982    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:09:01.888667 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:09:01.888632    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:09:08.433772 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:09:08.433735    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:09:08.525365 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:09:08.525331    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:09:08.711477 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:09:08.711389    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:09:08.867644 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:09:08.867612    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:09:15.093563 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:09:15.093522    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:09:15.196927 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:09:15.196893    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:09:22.134366 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:09:22.134330    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:09:22.571940 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:09:22.571859    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:09:22.771215 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:09:22.771182    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:09:25.779249 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:09:25.779218    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:09:25.882166 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:09:25.882128    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:09:25.960902 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:09:25.960870    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-66b55d3d-66cc45d47-xq8b8"]
Jul 23 12:09:25.961311 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:09:25.961293    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="73d2a882-e250-49b1-9fad-ff09626f22f3" containerName="maas-api"
Jul 23 12:09:25.961401 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:09:25.961313    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="73d2a882-e250-49b1-9fad-ff09626f22f3" containerName="maas-api"
Jul 23 12:09:25.961455 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:09:25.961428    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="73d2a882-e250-49b1-9fad-ff09626f22f3" containerName="maas-api"
Jul 23 12:09:25.964558 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:09:25.964537    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-66b55d3d-66cc45d47-xq8b8"
Jul 23 12:09:25.968387 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:09:25.968366    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-66b55d3d-serving-cert\""
Jul 23 12:09:25.982428 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:09:25.982398    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-66b55d3d-66cc45d47-xq8b8"]
Jul 23 12:09:26.010668 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:09:26.010636    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:09:26.108573 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:09:26.108480    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjp7j\" (UniqueName: \"kubernetes.io/projected/40d78ce3-754c-4758-b7fc-868e70aad8d7-kube-api-access-tjp7j\") pod \"maas-api-e2e-mt-66b55d3d-66cc45d47-xq8b8\" (UID: \"40d78ce3-754c-4758-b7fc-868e70aad8d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-66b55d3d-66cc45d47-xq8b8"
Jul 23 12:09:26.108573 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:09:26.108550    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/40d78ce3-754c-4758-b7fc-868e70aad8d7-openshift-service-ca\") pod \"maas-api-e2e-mt-66b55d3d-66cc45d47-xq8b8\" (UID: \"40d78ce3-754c-4758-b7fc-868e70aad8d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-66b55d3d-66cc45d47-xq8b8"
Jul 23 12:09:26.108843 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:09:26.108655    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/40d78ce3-754c-4758-b7fc-868e70aad8d7-maas-api-tls\") pod \"maas-api-e2e-mt-66b55d3d-66cc45d47-xq8b8\" (UID: \"40d78ce3-754c-4758-b7fc-868e70aad8d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-66b55d3d-66cc45d47-xq8b8"
Jul 23 12:09:26.209781 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:09:26.209732    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjp7j\" (UniqueName: \"kubernetes.io/projected/40d78ce3-754c-4758-b7fc-868e70aad8d7-kube-api-access-tjp7j\") pod \"maas-api-e2e-mt-66b55d3d-66cc45d47-xq8b8\" (UID: \"40d78ce3-754c-4758-b7fc-868e70aad8d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-66b55d3d-66cc45d47-xq8b8"
Jul 23 12:09:26.210017 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:09:26.209993    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/40d78ce3-754c-4758-b7fc-868e70aad8d7-openshift-service-ca\") pod \"maas-api-e2e-mt-66b55d3d-66cc45d47-xq8b8\" (UID: \"40d78ce3-754c-4758-b7fc-868e70aad8d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-66b55d3d-66cc45d47-xq8b8"
Jul 23 12:09:26.210115 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:09:26.210056    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/40d78ce3-754c-4758-b7fc-868e70aad8d7-maas-api-tls\") pod \"maas-api-e2e-mt-66b55d3d-66cc45d47-xq8b8\" (UID: \"40d78ce3-754c-4758-b7fc-868e70aad8d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-66b55d3d-66cc45d47-xq8b8"
Jul 23 12:09:26.210554 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:09:26.210536    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/40d78ce3-754c-4758-b7fc-868e70aad8d7-openshift-service-ca\") pod \"maas-api-e2e-mt-66b55d3d-66cc45d47-xq8b8\" (UID: \"40d78ce3-754c-4758-b7fc-868e70aad8d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-66b55d3d-66cc45d47-xq8b8"
Jul 23 12:09:26.212669 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:09:26.212653    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/40d78ce3-754c-4758-b7fc-868e70aad8d7-maas-api-tls\") pod \"maas-api-e2e-mt-66b55d3d-66cc45d47-xq8b8\" (UID: \"40d78ce3-754c-4758-b7fc-868e70aad8d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-66b55d3d-66cc45d47-xq8b8"
Jul 23 12:09:26.219231 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:09:26.219195    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:09:26.222142 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:09:26.222119    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjp7j\" (UniqueName: \"kubernetes.io/projected/40d78ce3-754c-4758-b7fc-868e70aad8d7-kube-api-access-tjp7j\") pod \"maas-api-e2e-mt-66b55d3d-66cc45d47-xq8b8\" (UID: \"40d78ce3-754c-4758-b7fc-868e70aad8d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-66b55d3d-66cc45d47-xq8b8"
Jul 23 12:09:26.275449 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:09:26.275414    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-66b55d3d-66cc45d47-xq8b8"
Jul 23 12:09:26.422755 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:09:26.422720    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-66b55d3d-66cc45d47-xq8b8"]
Jul 23 12:09:26.425621 ip-10-0-141-229 kubenswrapper[2563]: W0723 12:09:26.425594    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40d78ce3_754c_4758_b7fc_868e70aad8d7.slice/crio-393984f7dacaf142de1eba99a45abe25de796d5cdeaa36adef3eb2d024e14fdd WatchSource:0}: Error finding container 393984f7dacaf142de1eba99a45abe25de796d5cdeaa36adef3eb2d024e14fdd: Status 404 returned error can't find the container with id 393984f7dacaf142de1eba99a45abe25de796d5cdeaa36adef3eb2d024e14fdd
Jul 23 12:09:26.426906 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:09:26.426890    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 23 12:09:26.574260 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:09:26.574226    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-66b55d3d-66cc45d47-xq8b8" event={"ID":"40d78ce3-754c-4758-b7fc-868e70aad8d7","Type":"ContainerStarted","Data":"393984f7dacaf142de1eba99a45abe25de796d5cdeaa36adef3eb2d024e14fdd"}
Jul 23 12:09:26.774512 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:09:26.774430    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:09:26.881364 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:09:26.881331    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:09:27.580404 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:09:27.580367    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-66b55d3d-66cc45d47-xq8b8" event={"ID":"40d78ce3-754c-4758-b7fc-868e70aad8d7","Type":"ContainerStarted","Data":"2427d5cc264702a61e17450a89ec6e9b494682d723203ddeff309263c23f9712"}
Jul 23 12:09:27.580673 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:09:27.580434    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-66b55d3d-66cc45d47-xq8b8"
Jul 23 12:09:27.602804 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:09:27.602733    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-66b55d3d-66cc45d47-xq8b8" podStartSLOduration=2.242937971 podStartE2EDuration="2.60271648s" totalImagesPullingTime="359.778509ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 12:09:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 12:09:27.601861661 +0000 UTC m=+2021.994799450" watchObservedRunningTime="2026-07-23 12:09:27.60271648 +0000 UTC m=+2021.995654268"
Jul 23 12:09:30.265899 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:09:30.265865    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:09:33.590575 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:09:33.590546    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-66b55d3d-66cc45d47-xq8b8"
Jul 23 12:10:05.712154 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:10:05.712119    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:10:46.150739 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:10:46.150705    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nhm9p_93649ac9-9062-46c0-adaa-1048d471211a/ovn-acl-logging/0.log"
Jul 23 12:10:46.155333 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:10:46.155311    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nhm9p_93649ac9-9062-46c0-adaa-1048d471211a/ovn-acl-logging/0.log"
Jul 23 12:10:51.767506 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:10:51.767473    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:10:51.867209 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:10:51.867175    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:10:55.306921 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:10:55.306888    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-66b55d3d-66cc45d47-xq8b8"]
Jul 23 12:10:55.307382 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:10:55.307134    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-66b55d3d-66cc45d47-xq8b8" podUID="40d78ce3-754c-4758-b7fc-868e70aad8d7" containerName="maas-api" containerID="cri-o://2427d5cc264702a61e17450a89ec6e9b494682d723203ddeff309263c23f9712" gracePeriod=30
Jul 23 12:10:55.333906 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:10:55.333879    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:10:55.555424 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:10:55.555400    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-66b55d3d-66cc45d47-xq8b8"
Jul 23 12:10:55.652780 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:10:55.652748    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tjp7j\" (UniqueName: \"kubernetes.io/projected/40d78ce3-754c-4758-b7fc-868e70aad8d7-kube-api-access-tjp7j\") pod \"40d78ce3-754c-4758-b7fc-868e70aad8d7\" (UID: \"40d78ce3-754c-4758-b7fc-868e70aad8d7\") "
Jul 23 12:10:55.652780 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:10:55.652781    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/40d78ce3-754c-4758-b7fc-868e70aad8d7-openshift-service-ca\") pod \"40d78ce3-754c-4758-b7fc-868e70aad8d7\" (UID: \"40d78ce3-754c-4758-b7fc-868e70aad8d7\") "
Jul 23 12:10:55.652992 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:10:55.652810    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/40d78ce3-754c-4758-b7fc-868e70aad8d7-maas-api-tls\") pod \"40d78ce3-754c-4758-b7fc-868e70aad8d7\" (UID: \"40d78ce3-754c-4758-b7fc-868e70aad8d7\") "
Jul 23 12:10:55.653128 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:10:55.653104    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40d78ce3-754c-4758-b7fc-868e70aad8d7-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "40d78ce3-754c-4758-b7fc-868e70aad8d7" (UID: "40d78ce3-754c-4758-b7fc-868e70aad8d7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 12:10:55.653222 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:10:55.653193    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/40d78ce3-754c-4758-b7fc-868e70aad8d7-openshift-service-ca\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 23 12:10:55.655076 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:10:55.655051    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40d78ce3-754c-4758-b7fc-868e70aad8d7-kube-api-access-tjp7j" (OuterVolumeSpecName: "kube-api-access-tjp7j") pod "40d78ce3-754c-4758-b7fc-868e70aad8d7" (UID: "40d78ce3-754c-4758-b7fc-868e70aad8d7"). InnerVolumeSpecName "kube-api-access-tjp7j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 12:10:55.655180 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:10:55.655079    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/40d78ce3-754c-4758-b7fc-868e70aad8d7-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "40d78ce3-754c-4758-b7fc-868e70aad8d7" (UID: "40d78ce3-754c-4758-b7fc-868e70aad8d7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 12:10:55.753832 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:10:55.753796    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tjp7j\" (UniqueName: \"kubernetes.io/projected/40d78ce3-754c-4758-b7fc-868e70aad8d7-kube-api-access-tjp7j\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 23 12:10:55.753832 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:10:55.753826    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/40d78ce3-754c-4758-b7fc-868e70aad8d7-maas-api-tls\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 23 12:10:55.997178 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:10:55.997089    2563 generic.go:356] "Generic (PLEG): container finished" podID="40d78ce3-754c-4758-b7fc-868e70aad8d7" containerID="2427d5cc264702a61e17450a89ec6e9b494682d723203ddeff309263c23f9712" exitCode=0
Jul 23 12:10:55.997178 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:10:55.997126    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-66b55d3d-66cc45d47-xq8b8" event={"ID":"40d78ce3-754c-4758-b7fc-868e70aad8d7","Type":"ContainerDied","Data":"2427d5cc264702a61e17450a89ec6e9b494682d723203ddeff309263c23f9712"}
Jul 23 12:10:55.997178 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:10:55.997138    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-66b55d3d-66cc45d47-xq8b8"
Jul 23 12:10:55.997178 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:10:55.997166    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-66b55d3d-66cc45d47-xq8b8" event={"ID":"40d78ce3-754c-4758-b7fc-868e70aad8d7","Type":"ContainerDied","Data":"393984f7dacaf142de1eba99a45abe25de796d5cdeaa36adef3eb2d024e14fdd"}
Jul 23 12:10:55.997473 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:10:55.997187    2563 scope.go:117] "RemoveContainer" containerID="2427d5cc264702a61e17450a89ec6e9b494682d723203ddeff309263c23f9712"
Jul 23 12:10:56.006087 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:10:56.006067    2563 scope.go:117] "RemoveContainer" containerID="2427d5cc264702a61e17450a89ec6e9b494682d723203ddeff309263c23f9712"
Jul 23 12:10:56.006336 ip-10-0-141-229 kubenswrapper[2563]: E0723 12:10:56.006318    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2427d5cc264702a61e17450a89ec6e9b494682d723203ddeff309263c23f9712\": container with ID starting with 2427d5cc264702a61e17450a89ec6e9b494682d723203ddeff309263c23f9712 not found: ID does not exist" containerID="2427d5cc264702a61e17450a89ec6e9b494682d723203ddeff309263c23f9712"
Jul 23 12:10:56.006384 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:10:56.006345    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2427d5cc264702a61e17450a89ec6e9b494682d723203ddeff309263c23f9712"} err="failed to get container status \"2427d5cc264702a61e17450a89ec6e9b494682d723203ddeff309263c23f9712\": rpc error: code = NotFound desc = could not find container \"2427d5cc264702a61e17450a89ec6e9b494682d723203ddeff309263c23f9712\": container with ID starting with 2427d5cc264702a61e17450a89ec6e9b494682d723203ddeff309263c23f9712 not found: ID does not exist"
Jul 23 12:10:56.018931 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:10:56.018911    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-66b55d3d-66cc45d47-xq8b8"]
Jul 23 12:10:56.022191 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:10:56.022170    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-66b55d3d-66cc45d47-xq8b8"]
Jul 23 12:10:56.078645 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:10:56.078613    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="40d78ce3-754c-4758-b7fc-868e70aad8d7" path="/var/lib/kubelet/pods/40d78ce3-754c-4758-b7fc-868e70aad8d7/volumes"
Jul 23 12:10:59.957991 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:10:59.957952    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:11:02.355031 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:11:02.354997    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:11:07.379547 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:11:07.379512    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:11:10.154861 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:11:10.154828    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:11:44.093438 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:11:44.093357    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:11:44.885364 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:11:44.885331    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:11:50.747157 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:11:50.747111    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:11:50.859032 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:11:50.858999    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:11:51.046038 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:11:51.045957    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:11:51.296883 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:11:51.296788    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:11:52.420333 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:11:52.420286    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:11:52.518184 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:11:52.518150    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:11:52.617666 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:11:52.617617    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:11:53.412762 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:11:53.412731    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:11:53.509615 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:11:53.509584    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:11:53.610308 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:11:53.610276    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:11:55.061498 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:11:55.061462    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:12:00.850361 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:12:00.850326    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:12:00.947820 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:12:00.947777    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:12:01.574876 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:12:01.574838    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:12:07.626467 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:12:07.626425    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:12:07.736501 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:12:07.736457    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:12:07.837499 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:12:07.837464    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:12:18.406650 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:12:18.406603    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:12:18.776172 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:12:18.776086    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:12:39.797946 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:12:39.797909    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:12:46.428781 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:12:46.428714    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:12:46.529643 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:12:46.529610    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:12:46.843921 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:12:46.843834    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:12:47.284825 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:12:47.284796    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:12:47.383703 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:12:47.383658    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:12:50.817805 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:12:50.817757    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:12:50.888390 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:12:50.888353    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-9d4f98-6bc7dc8dd6-2rd2g"]
Jul 23 12:12:50.889010 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:12:50.888985    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="40d78ce3-754c-4758-b7fc-868e70aad8d7" containerName="maas-api"
Jul 23 12:12:50.889010 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:12:50.889008    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="40d78ce3-754c-4758-b7fc-868e70aad8d7" containerName="maas-api"
Jul 23 12:12:50.889164 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:12:50.889099    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="40d78ce3-754c-4758-b7fc-868e70aad8d7" containerName="maas-api"
Jul 23 12:12:50.892480 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:12:50.892455    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9d4f98-6bc7dc8dd6-2rd2g"
Jul 23 12:12:50.895365 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:12:50.895342    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-9d4f98-serving-cert\""
Jul 23 12:12:50.905337 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:12:50.905305    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-9d4f98-6bc7dc8dd6-2rd2g"]
Jul 23 12:12:50.910476 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:12:50.910448    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:12:50.986205 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:12:50.986165    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:12:51.010521 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:12:51.010486    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ac4f6c6-e7f3-43c8-8331-2378d9c94a9f-openshift-service-ca\") pod \"maas-api-e2e-inf-a-9d4f98-6bc7dc8dd6-2rd2g\" (UID: \"9ac4f6c6-e7f3-43c8-8331-2378d9c94a9f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9d4f98-6bc7dc8dd6-2rd2g"
Jul 23 12:12:51.010727 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:12:51.010559    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9ac4f6c6-e7f3-43c8-8331-2378d9c94a9f-maas-api-tls\") pod \"maas-api-e2e-inf-a-9d4f98-6bc7dc8dd6-2rd2g\" (UID: \"9ac4f6c6-e7f3-43c8-8331-2378d9c94a9f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9d4f98-6bc7dc8dd6-2rd2g"
Jul 23 12:12:51.010727 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:12:51.010677    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-splrv\" (UniqueName: \"kubernetes.io/projected/9ac4f6c6-e7f3-43c8-8331-2378d9c94a9f-kube-api-access-splrv\") pod \"maas-api-e2e-inf-a-9d4f98-6bc7dc8dd6-2rd2g\" (UID: \"9ac4f6c6-e7f3-43c8-8331-2378d9c94a9f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9d4f98-6bc7dc8dd6-2rd2g"
Jul 23 12:12:51.089013 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:12:51.088913    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:12:51.111794 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:12:51.111748    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-splrv\" (UniqueName: \"kubernetes.io/projected/9ac4f6c6-e7f3-43c8-8331-2378d9c94a9f-kube-api-access-splrv\") pod \"maas-api-e2e-inf-a-9d4f98-6bc7dc8dd6-2rd2g\" (UID: \"9ac4f6c6-e7f3-43c8-8331-2378d9c94a9f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9d4f98-6bc7dc8dd6-2rd2g"
Jul 23 12:12:51.111987 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:12:51.111819    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ac4f6c6-e7f3-43c8-8331-2378d9c94a9f-openshift-service-ca\") pod \"maas-api-e2e-inf-a-9d4f98-6bc7dc8dd6-2rd2g\" (UID: \"9ac4f6c6-e7f3-43c8-8331-2378d9c94a9f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9d4f98-6bc7dc8dd6-2rd2g"
Jul 23 12:12:51.111987 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:12:51.111881    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9ac4f6c6-e7f3-43c8-8331-2378d9c94a9f-maas-api-tls\") pod \"maas-api-e2e-inf-a-9d4f98-6bc7dc8dd6-2rd2g\" (UID: \"9ac4f6c6-e7f3-43c8-8331-2378d9c94a9f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9d4f98-6bc7dc8dd6-2rd2g"
Jul 23 12:12:51.112438 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:12:51.112403    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ac4f6c6-e7f3-43c8-8331-2378d9c94a9f-openshift-service-ca\") pod \"maas-api-e2e-inf-a-9d4f98-6bc7dc8dd6-2rd2g\" (UID: \"9ac4f6c6-e7f3-43c8-8331-2378d9c94a9f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9d4f98-6bc7dc8dd6-2rd2g"
Jul 23 12:12:51.114571 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:12:51.114543    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9ac4f6c6-e7f3-43c8-8331-2378d9c94a9f-maas-api-tls\") pod \"maas-api-e2e-inf-a-9d4f98-6bc7dc8dd6-2rd2g\" (UID: \"9ac4f6c6-e7f3-43c8-8331-2378d9c94a9f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9d4f98-6bc7dc8dd6-2rd2g"
Jul 23 12:12:51.120375 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:12:51.120344    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-splrv\" (UniqueName: \"kubernetes.io/projected/9ac4f6c6-e7f3-43c8-8331-2378d9c94a9f-kube-api-access-splrv\") pod \"maas-api-e2e-inf-a-9d4f98-6bc7dc8dd6-2rd2g\" (UID: \"9ac4f6c6-e7f3-43c8-8331-2378d9c94a9f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9d4f98-6bc7dc8dd6-2rd2g"
Jul 23 12:12:51.203479 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:12:51.203442    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9d4f98-6bc7dc8dd6-2rd2g"
Jul 23 12:12:51.351998 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:12:51.351913    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-9d4f98-6bc7dc8dd6-2rd2g"]
Jul 23 12:12:51.354439 ip-10-0-141-229 kubenswrapper[2563]: W0723 12:12:51.354409    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ac4f6c6_e7f3_43c8_8331_2378d9c94a9f.slice/crio-7a8611df4d8c943ee313032e4d277a4e904857f4251d866e9e0603e9c7d29435 WatchSource:0}: Error finding container 7a8611df4d8c943ee313032e4d277a4e904857f4251d866e9e0603e9c7d29435: Status 404 returned error can't find the container with id 7a8611df4d8c943ee313032e4d277a4e904857f4251d866e9e0603e9c7d29435
Jul 23 12:12:51.476326 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:12:51.476286    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:12:51.536872 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:12:51.536836    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9d4f98-6bc7dc8dd6-2rd2g" event={"ID":"9ac4f6c6-e7f3-43c8-8331-2378d9c94a9f","Type":"ContainerStarted","Data":"7a8611df4d8c943ee313032e4d277a4e904857f4251d866e9e0603e9c7d29435"}
Jul 23 12:12:51.575979 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:12:51.575950    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:12:52.542538 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:12:52.542496    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9d4f98-6bc7dc8dd6-2rd2g" event={"ID":"9ac4f6c6-e7f3-43c8-8331-2378d9c94a9f","Type":"ContainerStarted","Data":"5d330769c0f3cf987b0d2164c787fc7e6cde073ddfe206fd3bb1483d4c7f717e"}
Jul 23 12:12:52.542979 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:12:52.542582    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9d4f98-6bc7dc8dd6-2rd2g"
Jul 23 12:12:52.560224 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:12:52.560162    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9d4f98-6bc7dc8dd6-2rd2g" podStartSLOduration=2.106124996 podStartE2EDuration="2.560143565s" totalImagesPullingTime="454.018569ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 12:12:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 12:12:52.558661829 +0000 UTC m=+2226.951599617" watchObservedRunningTime="2026-07-23 12:12:52.560143565 +0000 UTC m=+2226.953081353"
Jul 23 12:12:55.365870 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:12:55.365809    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:12:55.707816 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:12:55.707698    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:12:56.124528 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:12:56.124493    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:12:56.753729 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:12:56.753662    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:12:58.555322 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:12:58.555289    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9d4f98-6bc7dc8dd6-2rd2g"
Jul 23 12:12:58.834786 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:12:58.834706    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:12:58.928547 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:12:58.928507    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:12:59.025889 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:12:59.025852    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:12:59.137505 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:12:59.137466    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:12:59.520453 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:12:59.520357    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:12:59.622493 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:12:59.622459    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:13:03.375779 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:13:03.375744    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:13:08.285988 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:13:08.285907    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:13:12.286166 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:13:12.286130    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:13:12.382544 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:13:12.382507    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:13:20.299750 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:13:20.299705    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:13:21.781706 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:13:21.781643    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:13:31.639728 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:13:31.639677    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-480488/test-model-480488-kserve-55d5b5585c-kl2gz"]
Jul 23 12:13:31.643326 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:13:31.643308    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-480488/test-model-480488-kserve-55d5b5585c-kl2gz"
Jul 23 12:13:31.645839 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:13:31.645815    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-480488\"/\"openshift-service-ca.crt\""
Jul 23 12:13:31.645959 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:13:31.645893    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-480488\"/\"test-model-480488-kserve-self-signed-certs\""
Jul 23 12:13:31.646660 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:13:31.646642    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-480488\"/\"kube-root-ca.crt\""
Jul 23 12:13:31.646813 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:13:31.646797    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-480488\"/\"default-dockercfg-7gkvs\""
Jul 23 12:13:31.651909 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:13:31.651885    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-480488/test-model-480488-kserve-55d5b5585c-kl2gz"]
Jul 23 12:13:31.666251 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:13:31.666213    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:13:31.705181 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:13:31.705141    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c486a743-ade9-4b78-a022-e2095d5381bc-tls-certs\") pod \"test-model-480488-kserve-55d5b5585c-kl2gz\" (UID: \"c486a743-ade9-4b78-a022-e2095d5381bc\") " pod="ai-tenant-e2e-inf-b-480488/test-model-480488-kserve-55d5b5585c-kl2gz"
Jul 23 12:13:31.705181 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:13:31.705182    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6556g\" (UniqueName: \"kubernetes.io/projected/c486a743-ade9-4b78-a022-e2095d5381bc-kube-api-access-6556g\") pod \"test-model-480488-kserve-55d5b5585c-kl2gz\" (UID: \"c486a743-ade9-4b78-a022-e2095d5381bc\") " pod="ai-tenant-e2e-inf-b-480488/test-model-480488-kserve-55d5b5585c-kl2gz"
Jul 23 12:13:31.705392 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:13:31.705295    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c486a743-ade9-4b78-a022-e2095d5381bc-home\") pod \"test-model-480488-kserve-55d5b5585c-kl2gz\" (UID: \"c486a743-ade9-4b78-a022-e2095d5381bc\") " pod="ai-tenant-e2e-inf-b-480488/test-model-480488-kserve-55d5b5585c-kl2gz"
Jul 23 12:13:31.705392 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:13:31.705340    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c486a743-ade9-4b78-a022-e2095d5381bc-dshm\") pod \"test-model-480488-kserve-55d5b5585c-kl2gz\" (UID: \"c486a743-ade9-4b78-a022-e2095d5381bc\") " pod="ai-tenant-e2e-inf-b-480488/test-model-480488-kserve-55d5b5585c-kl2gz"
Jul 23 12:13:31.705392 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:13:31.705358    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c486a743-ade9-4b78-a022-e2095d5381bc-model-cache\") pod \"test-model-480488-kserve-55d5b5585c-kl2gz\" (UID: \"c486a743-ade9-4b78-a022-e2095d5381bc\") " pod="ai-tenant-e2e-inf-b-480488/test-model-480488-kserve-55d5b5585c-kl2gz"
Jul 23 12:13:31.705392 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:13:31.705374    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c486a743-ade9-4b78-a022-e2095d5381bc-tmp-dir\") pod \"test-model-480488-kserve-55d5b5585c-kl2gz\" (UID: \"c486a743-ade9-4b78-a022-e2095d5381bc\") " pod="ai-tenant-e2e-inf-b-480488/test-model-480488-kserve-55d5b5585c-kl2gz"
Jul 23 12:13:31.806526 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:13:31.806471    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c486a743-ade9-4b78-a022-e2095d5381bc-home\") pod \"test-model-480488-kserve-55d5b5585c-kl2gz\" (UID: \"c486a743-ade9-4b78-a022-e2095d5381bc\") " pod="ai-tenant-e2e-inf-b-480488/test-model-480488-kserve-55d5b5585c-kl2gz"
Jul 23 12:13:31.806760 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:13:31.806539    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c486a743-ade9-4b78-a022-e2095d5381bc-dshm\") pod \"test-model-480488-kserve-55d5b5585c-kl2gz\" (UID: \"c486a743-ade9-4b78-a022-e2095d5381bc\") " pod="ai-tenant-e2e-inf-b-480488/test-model-480488-kserve-55d5b5585c-kl2gz"
Jul 23 12:13:31.806760 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:13:31.806558    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c486a743-ade9-4b78-a022-e2095d5381bc-model-cache\") pod \"test-model-480488-kserve-55d5b5585c-kl2gz\" (UID: \"c486a743-ade9-4b78-a022-e2095d5381bc\") " pod="ai-tenant-e2e-inf-b-480488/test-model-480488-kserve-55d5b5585c-kl2gz"
Jul 23 12:13:31.806760 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:13:31.806574    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c486a743-ade9-4b78-a022-e2095d5381bc-tmp-dir\") pod \"test-model-480488-kserve-55d5b5585c-kl2gz\" (UID: \"c486a743-ade9-4b78-a022-e2095d5381bc\") " pod="ai-tenant-e2e-inf-b-480488/test-model-480488-kserve-55d5b5585c-kl2gz"
Jul 23 12:13:31.806760 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:13:31.806626    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c486a743-ade9-4b78-a022-e2095d5381bc-tls-certs\") pod \"test-model-480488-kserve-55d5b5585c-kl2gz\" (UID: \"c486a743-ade9-4b78-a022-e2095d5381bc\") " pod="ai-tenant-e2e-inf-b-480488/test-model-480488-kserve-55d5b5585c-kl2gz"
Jul 23 12:13:31.806760 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:13:31.806642    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6556g\" (UniqueName: \"kubernetes.io/projected/c486a743-ade9-4b78-a022-e2095d5381bc-kube-api-access-6556g\") pod \"test-model-480488-kserve-55d5b5585c-kl2gz\" (UID: \"c486a743-ade9-4b78-a022-e2095d5381bc\") " pod="ai-tenant-e2e-inf-b-480488/test-model-480488-kserve-55d5b5585c-kl2gz"
Jul 23 12:13:31.807039 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:13:31.806987    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c486a743-ade9-4b78-a022-e2095d5381bc-home\") pod \"test-model-480488-kserve-55d5b5585c-kl2gz\" (UID: \"c486a743-ade9-4b78-a022-e2095d5381bc\") " pod="ai-tenant-e2e-inf-b-480488/test-model-480488-kserve-55d5b5585c-kl2gz"
Jul 23 12:13:31.807039 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:13:31.807012    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c486a743-ade9-4b78-a022-e2095d5381bc-tmp-dir\") pod \"test-model-480488-kserve-55d5b5585c-kl2gz\" (UID: \"c486a743-ade9-4b78-a022-e2095d5381bc\") " pod="ai-tenant-e2e-inf-b-480488/test-model-480488-kserve-55d5b5585c-kl2gz"
Jul 23 12:13:31.807146 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:13:31.807054    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c486a743-ade9-4b78-a022-e2095d5381bc-model-cache\") pod \"test-model-480488-kserve-55d5b5585c-kl2gz\" (UID: \"c486a743-ade9-4b78-a022-e2095d5381bc\") " pod="ai-tenant-e2e-inf-b-480488/test-model-480488-kserve-55d5b5585c-kl2gz"
Jul 23 12:13:31.809167 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:13:31.809134    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c486a743-ade9-4b78-a022-e2095d5381bc-dshm\") pod \"test-model-480488-kserve-55d5b5585c-kl2gz\" (UID: \"c486a743-ade9-4b78-a022-e2095d5381bc\") " pod="ai-tenant-e2e-inf-b-480488/test-model-480488-kserve-55d5b5585c-kl2gz"
Jul 23 12:13:31.809400 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:13:31.809380    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c486a743-ade9-4b78-a022-e2095d5381bc-tls-certs\") pod \"test-model-480488-kserve-55d5b5585c-kl2gz\" (UID: \"c486a743-ade9-4b78-a022-e2095d5381bc\") " pod="ai-tenant-e2e-inf-b-480488/test-model-480488-kserve-55d5b5585c-kl2gz"
Jul 23 12:13:31.815306 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:13:31.815285    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6556g\" (UniqueName: \"kubernetes.io/projected/c486a743-ade9-4b78-a022-e2095d5381bc-kube-api-access-6556g\") pod \"test-model-480488-kserve-55d5b5585c-kl2gz\" (UID: \"c486a743-ade9-4b78-a022-e2095d5381bc\") " pod="ai-tenant-e2e-inf-b-480488/test-model-480488-kserve-55d5b5585c-kl2gz"
Jul 23 12:13:31.953768 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:13:31.953626    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-480488/test-model-480488-kserve-55d5b5585c-kl2gz"
Jul 23 12:13:32.099205 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:13:32.099178    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-480488/test-model-480488-kserve-55d5b5585c-kl2gz"]
Jul 23 12:13:32.104522 ip-10-0-141-229 kubenswrapper[2563]: W0723 12:13:32.101820    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc486a743_ade9_4b78_a022_e2095d5381bc.slice/crio-90269cbd2fa2d5237723cc007f79499da2517fdb51c36f4035ba4a1dc09dfcf8 WatchSource:0}: Error finding container 90269cbd2fa2d5237723cc007f79499da2517fdb51c36f4035ba4a1dc09dfcf8: Status 404 returned error can't find the container with id 90269cbd2fa2d5237723cc007f79499da2517fdb51c36f4035ba4a1dc09dfcf8
Jul 23 12:13:32.746061 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:13:32.746024    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-480488/test-model-480488-kserve-55d5b5585c-kl2gz" event={"ID":"c486a743-ade9-4b78-a022-e2095d5381bc","Type":"ContainerStarted","Data":"1fd7e42ddf7d8733a2f862c2f21cd03d1c8b124cb7a7248ab379ebb2a8b6b9fb"}
Jul 23 12:13:32.746061 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:13:32.746068    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-480488/test-model-480488-kserve-55d5b5585c-kl2gz" event={"ID":"c486a743-ade9-4b78-a022-e2095d5381bc","Type":"ContainerStarted","Data":"90269cbd2fa2d5237723cc007f79499da2517fdb51c36f4035ba4a1dc09dfcf8"}
Jul 23 12:13:32.765463 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:13:32.765397    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-480488/test-model-480488-kserve-55d5b5585c-kl2gz" podStartSLOduration=1.765379745 podStartE2EDuration="1.765379745s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 12:13:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 12:13:32.763344735 +0000 UTC m=+2267.156282523" watchObservedRunningTime="2026-07-23 12:13:32.765379745 +0000 UTC m=+2267.158317532"
Jul 23 12:13:41.954735 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:13:41.954696    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-480488/test-model-480488-kserve-55d5b5585c-kl2gz"
Jul 23 12:13:41.954735 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:13:41.954747    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-480488/test-model-480488-kserve-55d5b5585c-kl2gz"
Jul 23 12:13:41.967853 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:13:41.967821    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-480488/test-model-480488-kserve-55d5b5585c-kl2gz"
Jul 23 12:13:42.808042 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:13:42.808000    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-480488/test-model-480488-kserve-55d5b5585c-kl2gz"
Jul 23 12:14:13.022667 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:14:13.022609    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:14:14.024094 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:14:14.024059    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-480488/test-model-480488-kserve-55d5b5585c-kl2gz"]
Jul 23 12:14:14.050483 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:14:14.050447    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:14:14.741406 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:14:14.741303    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:14:14.935405 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:14:14.935374    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:14:24.680304 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:14:24.680269    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-9d4f98-6bc7dc8dd6-2rd2g"]
Jul 23 12:14:24.681016 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:14:24.680978    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9d4f98-6bc7dc8dd6-2rd2g" podUID="9ac4f6c6-e7f3-43c8-8331-2378d9c94a9f" containerName="maas-api" containerID="cri-o://5d330769c0f3cf987b0d2164c787fc7e6cde073ddfe206fd3bb1483d4c7f717e" gracePeriod=30
Jul 23 12:14:24.716605 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:14:24.716556    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:14:24.941122 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:14:24.941043    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9d4f98-6bc7dc8dd6-2rd2g"
Jul 23 12:14:25.003939 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:14:25.003901    2563 generic.go:356] "Generic (PLEG): container finished" podID="9ac4f6c6-e7f3-43c8-8331-2378d9c94a9f" containerID="5d330769c0f3cf987b0d2164c787fc7e6cde073ddfe206fd3bb1483d4c7f717e" exitCode=0
Jul 23 12:14:25.004164 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:14:25.003956    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9d4f98-6bc7dc8dd6-2rd2g"
Jul 23 12:14:25.004164 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:14:25.003971    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9d4f98-6bc7dc8dd6-2rd2g" event={"ID":"9ac4f6c6-e7f3-43c8-8331-2378d9c94a9f","Type":"ContainerDied","Data":"5d330769c0f3cf987b0d2164c787fc7e6cde073ddfe206fd3bb1483d4c7f717e"}
Jul 23 12:14:25.004164 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:14:25.004071    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9d4f98-6bc7dc8dd6-2rd2g" event={"ID":"9ac4f6c6-e7f3-43c8-8331-2378d9c94a9f","Type":"ContainerDied","Data":"7a8611df4d8c943ee313032e4d277a4e904857f4251d866e9e0603e9c7d29435"}
Jul 23 12:14:25.004164 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:14:25.004102    2563 scope.go:117] "RemoveContainer" containerID="5d330769c0f3cf987b0d2164c787fc7e6cde073ddfe206fd3bb1483d4c7f717e"
Jul 23 12:14:25.013041 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:14:25.013017    2563 scope.go:117] "RemoveContainer" containerID="5d330769c0f3cf987b0d2164c787fc7e6cde073ddfe206fd3bb1483d4c7f717e"
Jul 23 12:14:25.013370 ip-10-0-141-229 kubenswrapper[2563]: E0723 12:14:25.013350    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d330769c0f3cf987b0d2164c787fc7e6cde073ddfe206fd3bb1483d4c7f717e\": container with ID starting with 5d330769c0f3cf987b0d2164c787fc7e6cde073ddfe206fd3bb1483d4c7f717e not found: ID does not exist" containerID="5d330769c0f3cf987b0d2164c787fc7e6cde073ddfe206fd3bb1483d4c7f717e"
Jul 23 12:14:25.013419 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:14:25.013380    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d330769c0f3cf987b0d2164c787fc7e6cde073ddfe206fd3bb1483d4c7f717e"} err="failed to get container status \"5d330769c0f3cf987b0d2164c787fc7e6cde073ddfe206fd3bb1483d4c7f717e\": rpc error: code = NotFound desc = could not find container \"5d330769c0f3cf987b0d2164c787fc7e6cde073ddfe206fd3bb1483d4c7f717e\": container with ID starting with 5d330769c0f3cf987b0d2164c787fc7e6cde073ddfe206fd3bb1483d4c7f717e not found: ID does not exist"
Jul 23 12:14:25.024504 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:14:25.024467    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9ac4f6c6-e7f3-43c8-8331-2378d9c94a9f-maas-api-tls\") pod \"9ac4f6c6-e7f3-43c8-8331-2378d9c94a9f\" (UID: \"9ac4f6c6-e7f3-43c8-8331-2378d9c94a9f\") "
Jul 23 12:14:25.024670 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:14:25.024531    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ac4f6c6-e7f3-43c8-8331-2378d9c94a9f-openshift-service-ca\") pod \"9ac4f6c6-e7f3-43c8-8331-2378d9c94a9f\" (UID: \"9ac4f6c6-e7f3-43c8-8331-2378d9c94a9f\") "
Jul 23 12:14:25.024670 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:14:25.024620    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-splrv\" (UniqueName: \"kubernetes.io/projected/9ac4f6c6-e7f3-43c8-8331-2378d9c94a9f-kube-api-access-splrv\") pod \"9ac4f6c6-e7f3-43c8-8331-2378d9c94a9f\" (UID: \"9ac4f6c6-e7f3-43c8-8331-2378d9c94a9f\") "
Jul 23 12:14:25.024961 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:14:25.024923    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ac4f6c6-e7f3-43c8-8331-2378d9c94a9f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "9ac4f6c6-e7f3-43c8-8331-2378d9c94a9f" (UID: "9ac4f6c6-e7f3-43c8-8331-2378d9c94a9f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 12:14:25.026851 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:14:25.026819    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9ac4f6c6-e7f3-43c8-8331-2378d9c94a9f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "9ac4f6c6-e7f3-43c8-8331-2378d9c94a9f" (UID: "9ac4f6c6-e7f3-43c8-8331-2378d9c94a9f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 12:14:25.027078 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:14:25.027060    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ac4f6c6-e7f3-43c8-8331-2378d9c94a9f-kube-api-access-splrv" (OuterVolumeSpecName: "kube-api-access-splrv") pod "9ac4f6c6-e7f3-43c8-8331-2378d9c94a9f" (UID: "9ac4f6c6-e7f3-43c8-8331-2378d9c94a9f"). InnerVolumeSpecName "kube-api-access-splrv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 12:14:25.125376 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:14:25.125330    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-splrv\" (UniqueName: \"kubernetes.io/projected/9ac4f6c6-e7f3-43c8-8331-2378d9c94a9f-kube-api-access-splrv\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 23 12:14:25.125376 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:14:25.125367    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9ac4f6c6-e7f3-43c8-8331-2378d9c94a9f-maas-api-tls\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 23 12:14:25.125376 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:14:25.125377    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ac4f6c6-e7f3-43c8-8331-2378d9c94a9f-openshift-service-ca\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 23 12:14:25.331941 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:14:25.331902    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-9d4f98-6bc7dc8dd6-2rd2g"]
Jul 23 12:14:25.340095 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:14:25.340059    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-9d4f98-6bc7dc8dd6-2rd2g"]
Jul 23 12:14:26.078332 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:14:26.078292    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9ac4f6c6-e7f3-43c8-8331-2378d9c94a9f" path="/var/lib/kubelet/pods/9ac4f6c6-e7f3-43c8-8331-2378d9c94a9f/volumes"
Jul 23 12:14:29.045440 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:14:29.045390    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-480488/test-model-480488-kserve-55d5b5585c-kl2gz" podUID="c486a743-ade9-4b78-a022-e2095d5381bc" containerName="main" containerID="cri-o://1fd7e42ddf7d8733a2f862c2f21cd03d1c8b124cb7a7248ab379ebb2a8b6b9fb" gracePeriod=45
Jul 23 12:14:29.269999 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:14:29.269963    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:14:29.305198 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:14:29.305129    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-480488/test-model-480488-kserve-55d5b5585c-kl2gz"
Jul 23 12:14:29.465434 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:14:29.465392    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c486a743-ade9-4b78-a022-e2095d5381bc-tmp-dir\") pod \"c486a743-ade9-4b78-a022-e2095d5381bc\" (UID: \"c486a743-ade9-4b78-a022-e2095d5381bc\") "
Jul 23 12:14:29.465629 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:14:29.465503    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c486a743-ade9-4b78-a022-e2095d5381bc-model-cache\") pod \"c486a743-ade9-4b78-a022-e2095d5381bc\" (UID: \"c486a743-ade9-4b78-a022-e2095d5381bc\") "
Jul 23 12:14:29.465629 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:14:29.465525    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c486a743-ade9-4b78-a022-e2095d5381bc-home\") pod \"c486a743-ade9-4b78-a022-e2095d5381bc\" (UID: \"c486a743-ade9-4b78-a022-e2095d5381bc\") "
Jul 23 12:14:29.465629 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:14:29.465559    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6556g\" (UniqueName: \"kubernetes.io/projected/c486a743-ade9-4b78-a022-e2095d5381bc-kube-api-access-6556g\") pod \"c486a743-ade9-4b78-a022-e2095d5381bc\" (UID: \"c486a743-ade9-4b78-a022-e2095d5381bc\") "
Jul 23 12:14:29.465629 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:14:29.465578    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c486a743-ade9-4b78-a022-e2095d5381bc-tls-certs\") pod \"c486a743-ade9-4b78-a022-e2095d5381bc\" (UID: \"c486a743-ade9-4b78-a022-e2095d5381bc\") "
Jul 23 12:14:29.465629 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:14:29.465596    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c486a743-ade9-4b78-a022-e2095d5381bc-dshm\") pod \"c486a743-ade9-4b78-a022-e2095d5381bc\" (UID: \"c486a743-ade9-4b78-a022-e2095d5381bc\") "
Jul 23 12:14:29.465961 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:14:29.465674    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c486a743-ade9-4b78-a022-e2095d5381bc-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c486a743-ade9-4b78-a022-e2095d5381bc" (UID: "c486a743-ade9-4b78-a022-e2095d5381bc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 12:14:29.465961 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:14:29.465800    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c486a743-ade9-4b78-a022-e2095d5381bc-model-cache" (OuterVolumeSpecName: "model-cache") pod "c486a743-ade9-4b78-a022-e2095d5381bc" (UID: "c486a743-ade9-4b78-a022-e2095d5381bc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 12:14:29.465961 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:14:29.465858    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c486a743-ade9-4b78-a022-e2095d5381bc-home" (OuterVolumeSpecName: "home") pod "c486a743-ade9-4b78-a022-e2095d5381bc" (UID: "c486a743-ade9-4b78-a022-e2095d5381bc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 12:14:29.466097 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:14:29.465971    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c486a743-ade9-4b78-a022-e2095d5381bc-model-cache\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 23 12:14:29.466097 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:14:29.465991    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c486a743-ade9-4b78-a022-e2095d5381bc-home\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 23 12:14:29.466097 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:14:29.466005    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c486a743-ade9-4b78-a022-e2095d5381bc-tmp-dir\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 23 12:14:29.468103 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:14:29.468076    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c486a743-ade9-4b78-a022-e2095d5381bc-dshm" (OuterVolumeSpecName: "dshm") pod "c486a743-ade9-4b78-a022-e2095d5381bc" (UID: "c486a743-ade9-4b78-a022-e2095d5381bc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 12:14:29.468103 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:14:29.468087    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c486a743-ade9-4b78-a022-e2095d5381bc-kube-api-access-6556g" (OuterVolumeSpecName: "kube-api-access-6556g") pod "c486a743-ade9-4b78-a022-e2095d5381bc" (UID: "c486a743-ade9-4b78-a022-e2095d5381bc"). InnerVolumeSpecName "kube-api-access-6556g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 12:14:29.468250 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:14:29.468145    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c486a743-ade9-4b78-a022-e2095d5381bc-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c486a743-ade9-4b78-a022-e2095d5381bc" (UID: "c486a743-ade9-4b78-a022-e2095d5381bc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 12:14:29.566575 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:14:29.566476    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c486a743-ade9-4b78-a022-e2095d5381bc-dshm\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 23 12:14:29.566575 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:14:29.566514    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6556g\" (UniqueName: \"kubernetes.io/projected/c486a743-ade9-4b78-a022-e2095d5381bc-kube-api-access-6556g\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 23 12:14:29.566575 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:14:29.566525    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c486a743-ade9-4b78-a022-e2095d5381bc-tls-certs\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 23 12:14:29.726011 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:14:29.725973    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:14:30.029159 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:14:30.029115    2563 generic.go:356] "Generic (PLEG): container finished" podID="c486a743-ade9-4b78-a022-e2095d5381bc" containerID="1fd7e42ddf7d8733a2f862c2f21cd03d1c8b124cb7a7248ab379ebb2a8b6b9fb" exitCode=0
Jul 23 12:14:30.029159 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:14:30.029164    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-480488/test-model-480488-kserve-55d5b5585c-kl2gz"
Jul 23 12:14:30.029387 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:14:30.029199    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-480488/test-model-480488-kserve-55d5b5585c-kl2gz" event={"ID":"c486a743-ade9-4b78-a022-e2095d5381bc","Type":"ContainerDied","Data":"1fd7e42ddf7d8733a2f862c2f21cd03d1c8b124cb7a7248ab379ebb2a8b6b9fb"}
Jul 23 12:14:30.029387 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:14:30.029245    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-480488/test-model-480488-kserve-55d5b5585c-kl2gz" event={"ID":"c486a743-ade9-4b78-a022-e2095d5381bc","Type":"ContainerDied","Data":"90269cbd2fa2d5237723cc007f79499da2517fdb51c36f4035ba4a1dc09dfcf8"}
Jul 23 12:14:30.029387 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:14:30.029262    2563 scope.go:117] "RemoveContainer" containerID="1fd7e42ddf7d8733a2f862c2f21cd03d1c8b124cb7a7248ab379ebb2a8b6b9fb"
Jul 23 12:14:30.043512 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:14:30.043489    2563 scope.go:117] "RemoveContainer" containerID="1fd7e42ddf7d8733a2f862c2f21cd03d1c8b124cb7a7248ab379ebb2a8b6b9fb"
Jul 23 12:14:30.043855 ip-10-0-141-229 kubenswrapper[2563]: E0723 12:14:30.043836    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1fd7e42ddf7d8733a2f862c2f21cd03d1c8b124cb7a7248ab379ebb2a8b6b9fb\": container with ID starting with 1fd7e42ddf7d8733a2f862c2f21cd03d1c8b124cb7a7248ab379ebb2a8b6b9fb not found: ID does not exist" containerID="1fd7e42ddf7d8733a2f862c2f21cd03d1c8b124cb7a7248ab379ebb2a8b6b9fb"
Jul 23 12:14:30.043933 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:14:30.043864    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1fd7e42ddf7d8733a2f862c2f21cd03d1c8b124cb7a7248ab379ebb2a8b6b9fb"} err="failed to get container status \"1fd7e42ddf7d8733a2f862c2f21cd03d1c8b124cb7a7248ab379ebb2a8b6b9fb\": rpc error: code = NotFound desc = could not find container \"1fd7e42ddf7d8733a2f862c2f21cd03d1c8b124cb7a7248ab379ebb2a8b6b9fb\": container with ID starting with 1fd7e42ddf7d8733a2f862c2f21cd03d1c8b124cb7a7248ab379ebb2a8b6b9fb not found: ID does not exist"
Jul 23 12:14:30.060765 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:14:30.060735    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-480488/test-model-480488-kserve-55d5b5585c-kl2gz"]
Jul 23 12:14:30.062508 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:14:30.062483    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-480488/test-model-480488-kserve-55d5b5585c-kl2gz"]
Jul 23 12:14:30.078260 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:14:30.078227    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c486a743-ade9-4b78-a022-e2095d5381bc" path="/var/lib/kubelet/pods/c486a743-ade9-4b78-a022-e2095d5381bc/volumes"
Jul 23 12:14:37.406846 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:14:37.406766    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:14:37.490107 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:14:37.490070    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:14:37.693418 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:14:37.693329    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:14:37.910452 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:14:37.910413    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:14:38.589903 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:14:38.589869    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:14:38.691375 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:14:38.691340    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:14:40.887040 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:14:40.886939    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:14:48.752348 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:14:48.752307    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:14:53.196873 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:14:53.196830    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:14:53.610133 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:14:53.610048    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:14:59.857178 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:14:59.857137    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:14:59.956597 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:14:59.956566    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:15:00.125042 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:15:00.125008    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:15:00.373052 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:15:00.373015    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:15:00.754896 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:15:00.754861    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:15:01.090169 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:15:01.090081    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:15:01.229401 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:15:01.229364    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:15:04.833843 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:15:04.833793    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:15:04.933410 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:15:04.933380    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:15:05.118283 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:15:05.118252    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:15:05.894561 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:15:05.894529    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:15:05.995098 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:15:05.995061    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:15:09.281522 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:15:09.281487    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:15:09.863941 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:15:09.863902    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:15:10.347430 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:15:10.347345    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:15:11.376331 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:15:11.376288    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:15:13.929325 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:15:13.929293    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:15:14.124372 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:15:14.124333    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:15:14.286326 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:15:14.286236    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:15:15.200434 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:15:15.200383    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:15:15.358219 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:15:15.358183    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:15:18.306944 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:15:18.306909    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:15:26.829707 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:15:26.829651    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:15:26.929107 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:15:26.929062    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:15:36.263530 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:15:36.263494    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:15:36.362112 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:15:36.362080    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:15:42.061488 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:15:42.061452    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:15:46.183987 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:15:46.183955    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nhm9p_93649ac9-9062-46c0-adaa-1048d471211a/ovn-acl-logging/0.log"
Jul 23 12:15:46.188812 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:15:46.188790    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nhm9p_93649ac9-9062-46c0-adaa-1048d471211a/ovn-acl-logging/0.log"
Jul 23 12:15:46.671747 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:15:46.671714    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:15:47.674378 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:15:47.674345    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:15:52.716483 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:15:52.716444    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:15:52.815264 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:15:52.815230    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:15:53.208484 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:15:53.208454    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:15:53.524593 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:15:53.524508    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:15:53.589191 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:15:53.589147    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:15:53.750538 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:15:53.750500    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:15:54.680250 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:15:54.680205    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:15:54.793390 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:15:54.793354    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:15:57.508858 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:15:57.508829    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:16:09.627961 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:16:09.627874    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:16:13.621872 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:16:13.621839    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:16:13.727559 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:16:13.727522    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:16:14.012585 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:16:14.012498    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:16:14.402010 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:16:14.401977    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:16:20.371660 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:16:20.371619    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:16:20.467784 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:16:20.467751    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:16:20.665504 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:16:20.665420    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:16:20.907026 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:16:20.906989    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:16:21.276775 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:16:21.276738    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:16:21.662854 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:16:21.662819    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:16:21.826443 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:16:21.826405    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:16:25.064160 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:16:25.064125    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:16:25.157200 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:16:25.157167    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:16:25.245662 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:16:25.245617    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:16:25.344575 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:16:25.344469    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:16:25.746791 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:16:25.746760    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:16:25.848564 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:16:25.848534    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:16:29.602322 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:16:29.602288    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:16:29.989427 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:16:29.989395    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:16:30.316639 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:16:30.316532    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:16:30.444226 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:16:30.444169    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:16:33.193652 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:16:33.193608    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:16:33.314051 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:16:33.314010    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:16:33.412897 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:16:33.412858    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:16:33.812209 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:16:33.812167    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:16:34.116112 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:16:34.116075    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:16:34.215423 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:16:34.215392    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:16:37.698063 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:16:37.698013    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:16:46.686172 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:16:46.686130    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:16:46.796157 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:16:46.796113    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:16:49.064003 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:16:49.063968    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-290648/auth-test-model-290648-kserve-6c5fc6967-hg22n"]
Jul 23 12:16:49.064423 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:16:49.064305    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9ac4f6c6-e7f3-43c8-8331-2378d9c94a9f" containerName="maas-api"
Jul 23 12:16:49.064423 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:16:49.064315    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="9ac4f6c6-e7f3-43c8-8331-2378d9c94a9f" containerName="maas-api"
Jul 23 12:16:49.064423 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:16:49.064338    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c486a743-ade9-4b78-a022-e2095d5381bc" containerName="main"
Jul 23 12:16:49.064423 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:16:49.064344    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="c486a743-ade9-4b78-a022-e2095d5381bc" containerName="main"
Jul 23 12:16:49.064423 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:16:49.064404    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="9ac4f6c6-e7f3-43c8-8331-2378d9c94a9f" containerName="maas-api"
Jul 23 12:16:49.064423 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:16:49.064413    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="c486a743-ade9-4b78-a022-e2095d5381bc" containerName="main"
Jul 23 12:16:49.067325 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:16:49.067304    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-290648/auth-test-model-290648-kserve-6c5fc6967-hg22n"
Jul 23 12:16:49.070320 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:16:49.070290    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-290648\"/\"default-dockercfg-4655d\""
Jul 23 12:16:49.070320 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:16:49.070302    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-290648\"/\"kube-root-ca.crt\""
Jul 23 12:16:49.070459 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:16:49.070290    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-290648\"/\"openshift-service-ca.crt\""
Jul 23 12:16:49.070459 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:16:49.070302    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-290648\"/\"auth-test-model-290648-kserve-self-signed-certs\""
Jul 23 12:16:49.077155 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:16:49.077133    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-290648/auth-test-model-290648-kserve-6c5fc6967-hg22n"]
Jul 23 12:16:49.087968 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:16:49.087941    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:16:49.167470 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:16:49.167435    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb0aa79d-41a3-49d7-95a7-4c4a96e2d89a-model-cache\") pod \"auth-test-model-290648-kserve-6c5fc6967-hg22n\" (UID: \"bb0aa79d-41a3-49d7-95a7-4c4a96e2d89a\") " pod="ai-tenant-e2e-shared-a-290648/auth-test-model-290648-kserve-6c5fc6967-hg22n"
Jul 23 12:16:49.167649 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:16:49.167477    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb0aa79d-41a3-49d7-95a7-4c4a96e2d89a-tmp-dir\") pod \"auth-test-model-290648-kserve-6c5fc6967-hg22n\" (UID: \"bb0aa79d-41a3-49d7-95a7-4c4a96e2d89a\") " pod="ai-tenant-e2e-shared-a-290648/auth-test-model-290648-kserve-6c5fc6967-hg22n"
Jul 23 12:16:49.167649 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:16:49.167550    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb0aa79d-41a3-49d7-95a7-4c4a96e2d89a-tls-certs\") pod \"auth-test-model-290648-kserve-6c5fc6967-hg22n\" (UID: \"bb0aa79d-41a3-49d7-95a7-4c4a96e2d89a\") " pod="ai-tenant-e2e-shared-a-290648/auth-test-model-290648-kserve-6c5fc6967-hg22n"
Jul 23 12:16:49.167649 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:16:49.167573    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bb0aa79d-41a3-49d7-95a7-4c4a96e2d89a-home\") pod \"auth-test-model-290648-kserve-6c5fc6967-hg22n\" (UID: \"bb0aa79d-41a3-49d7-95a7-4c4a96e2d89a\") " pod="ai-tenant-e2e-shared-a-290648/auth-test-model-290648-kserve-6c5fc6967-hg22n"
Jul 23 12:16:49.167649 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:16:49.167590    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dsb6v\" (UniqueName: \"kubernetes.io/projected/bb0aa79d-41a3-49d7-95a7-4c4a96e2d89a-kube-api-access-dsb6v\") pod \"auth-test-model-290648-kserve-6c5fc6967-hg22n\" (UID: \"bb0aa79d-41a3-49d7-95a7-4c4a96e2d89a\") " pod="ai-tenant-e2e-shared-a-290648/auth-test-model-290648-kserve-6c5fc6967-hg22n"
Jul 23 12:16:49.167649 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:16:49.167614    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb0aa79d-41a3-49d7-95a7-4c4a96e2d89a-dshm\") pod \"auth-test-model-290648-kserve-6c5fc6967-hg22n\" (UID: \"bb0aa79d-41a3-49d7-95a7-4c4a96e2d89a\") " pod="ai-tenant-e2e-shared-a-290648/auth-test-model-290648-kserve-6c5fc6967-hg22n"
Jul 23 12:16:49.268807 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:16:49.268766    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb0aa79d-41a3-49d7-95a7-4c4a96e2d89a-model-cache\") pod \"auth-test-model-290648-kserve-6c5fc6967-hg22n\" (UID: \"bb0aa79d-41a3-49d7-95a7-4c4a96e2d89a\") " pod="ai-tenant-e2e-shared-a-290648/auth-test-model-290648-kserve-6c5fc6967-hg22n"
Jul 23 12:16:49.268807 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:16:49.268809    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb0aa79d-41a3-49d7-95a7-4c4a96e2d89a-tmp-dir\") pod \"auth-test-model-290648-kserve-6c5fc6967-hg22n\" (UID: \"bb0aa79d-41a3-49d7-95a7-4c4a96e2d89a\") " pod="ai-tenant-e2e-shared-a-290648/auth-test-model-290648-kserve-6c5fc6967-hg22n"
Jul 23 12:16:49.269057 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:16:49.268865    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb0aa79d-41a3-49d7-95a7-4c4a96e2d89a-tls-certs\") pod \"auth-test-model-290648-kserve-6c5fc6967-hg22n\" (UID: \"bb0aa79d-41a3-49d7-95a7-4c4a96e2d89a\") " pod="ai-tenant-e2e-shared-a-290648/auth-test-model-290648-kserve-6c5fc6967-hg22n"
Jul 23 12:16:49.269057 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:16:49.268886    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bb0aa79d-41a3-49d7-95a7-4c4a96e2d89a-home\") pod \"auth-test-model-290648-kserve-6c5fc6967-hg22n\" (UID: \"bb0aa79d-41a3-49d7-95a7-4c4a96e2d89a\") " pod="ai-tenant-e2e-shared-a-290648/auth-test-model-290648-kserve-6c5fc6967-hg22n"
Jul 23 12:16:49.269057 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:16:49.268902    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dsb6v\" (UniqueName: \"kubernetes.io/projected/bb0aa79d-41a3-49d7-95a7-4c4a96e2d89a-kube-api-access-dsb6v\") pod \"auth-test-model-290648-kserve-6c5fc6967-hg22n\" (UID: \"bb0aa79d-41a3-49d7-95a7-4c4a96e2d89a\") " pod="ai-tenant-e2e-shared-a-290648/auth-test-model-290648-kserve-6c5fc6967-hg22n"
Jul 23 12:16:49.269057 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:16:49.268931    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb0aa79d-41a3-49d7-95a7-4c4a96e2d89a-dshm\") pod \"auth-test-model-290648-kserve-6c5fc6967-hg22n\" (UID: \"bb0aa79d-41a3-49d7-95a7-4c4a96e2d89a\") " pod="ai-tenant-e2e-shared-a-290648/auth-test-model-290648-kserve-6c5fc6967-hg22n"
Jul 23 12:16:49.269288 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:16:49.269192    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb0aa79d-41a3-49d7-95a7-4c4a96e2d89a-model-cache\") pod \"auth-test-model-290648-kserve-6c5fc6967-hg22n\" (UID: \"bb0aa79d-41a3-49d7-95a7-4c4a96e2d89a\") " pod="ai-tenant-e2e-shared-a-290648/auth-test-model-290648-kserve-6c5fc6967-hg22n"
Jul 23 12:16:49.269288 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:16:49.269251    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bb0aa79d-41a3-49d7-95a7-4c4a96e2d89a-home\") pod \"auth-test-model-290648-kserve-6c5fc6967-hg22n\" (UID: \"bb0aa79d-41a3-49d7-95a7-4c4a96e2d89a\") " pod="ai-tenant-e2e-shared-a-290648/auth-test-model-290648-kserve-6c5fc6967-hg22n"
Jul 23 12:16:49.269369 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:16:49.269293    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb0aa79d-41a3-49d7-95a7-4c4a96e2d89a-tmp-dir\") pod \"auth-test-model-290648-kserve-6c5fc6967-hg22n\" (UID: \"bb0aa79d-41a3-49d7-95a7-4c4a96e2d89a\") " pod="ai-tenant-e2e-shared-a-290648/auth-test-model-290648-kserve-6c5fc6967-hg22n"
Jul 23 12:16:49.271276 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:16:49.271247    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb0aa79d-41a3-49d7-95a7-4c4a96e2d89a-dshm\") pod \"auth-test-model-290648-kserve-6c5fc6967-hg22n\" (UID: \"bb0aa79d-41a3-49d7-95a7-4c4a96e2d89a\") " pod="ai-tenant-e2e-shared-a-290648/auth-test-model-290648-kserve-6c5fc6967-hg22n"
Jul 23 12:16:49.271460 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:16:49.271443    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb0aa79d-41a3-49d7-95a7-4c4a96e2d89a-tls-certs\") pod \"auth-test-model-290648-kserve-6c5fc6967-hg22n\" (UID: \"bb0aa79d-41a3-49d7-95a7-4c4a96e2d89a\") " pod="ai-tenant-e2e-shared-a-290648/auth-test-model-290648-kserve-6c5fc6967-hg22n"
Jul 23 12:16:49.277010 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:16:49.276983    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dsb6v\" (UniqueName: \"kubernetes.io/projected/bb0aa79d-41a3-49d7-95a7-4c4a96e2d89a-kube-api-access-dsb6v\") pod \"auth-test-model-290648-kserve-6c5fc6967-hg22n\" (UID: \"bb0aa79d-41a3-49d7-95a7-4c4a96e2d89a\") " pod="ai-tenant-e2e-shared-a-290648/auth-test-model-290648-kserve-6c5fc6967-hg22n"
Jul 23 12:16:49.377236 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:16:49.377203    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-290648/auth-test-model-290648-kserve-6c5fc6967-hg22n"
Jul 23 12:16:49.513600 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:16:49.513567    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-290648/auth-test-model-290648-kserve-6c5fc6967-hg22n"]
Jul 23 12:16:49.515864 ip-10-0-141-229 kubenswrapper[2563]: W0723 12:16:49.515833    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbb0aa79d_41a3_49d7_95a7_4c4a96e2d89a.slice/crio-c48565de2d365522e00f1231c60e7c961e01bf5b5c2bde714b34b0e578cfdabd WatchSource:0}: Error finding container c48565de2d365522e00f1231c60e7c961e01bf5b5c2bde714b34b0e578cfdabd: Status 404 returned error can't find the container with id c48565de2d365522e00f1231c60e7c961e01bf5b5c2bde714b34b0e578cfdabd
Jul 23 12:16:49.671936 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:16:49.671837    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-290648/auth-test-model-290648-kserve-6c5fc6967-hg22n" event={"ID":"bb0aa79d-41a3-49d7-95a7-4c4a96e2d89a","Type":"ContainerStarted","Data":"dcfc8482d1c505698138d05a746ebc40172040d8ce8f69b3a87fe0cf8391b703"}
Jul 23 12:16:49.671936 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:16:49.671888    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-290648/auth-test-model-290648-kserve-6c5fc6967-hg22n" event={"ID":"bb0aa79d-41a3-49d7-95a7-4c4a96e2d89a","Type":"ContainerStarted","Data":"c48565de2d365522e00f1231c60e7c961e01bf5b5c2bde714b34b0e578cfdabd"}
Jul 23 12:16:49.700383 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:16:49.700332    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-290648/auth-test-model-290648-kserve-6c5fc6967-hg22n" podStartSLOduration=0.700315911 podStartE2EDuration="700.315911ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 12:16:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 12:16:49.698490177 +0000 UTC m=+2464.091427964" watchObservedRunningTime="2026-07-23 12:16:49.700315911 +0000 UTC m=+2464.093253697"
Jul 23 12:16:55.629293 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:16:55.629247    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:16:55.731262 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:16:55.731220    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:16:59.378007 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:16:59.377968    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-290648/auth-test-model-290648-kserve-6c5fc6967-hg22n"
Jul 23 12:16:59.378491 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:16:59.378021    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-290648/auth-test-model-290648-kserve-6c5fc6967-hg22n"
Jul 23 12:16:59.390463 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:16:59.390432    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-290648/auth-test-model-290648-kserve-6c5fc6967-hg22n"
Jul 23 12:16:59.728081 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:16:59.727997    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-290648/auth-test-model-290648-kserve-6c5fc6967-hg22n"
Jul 23 12:17:02.530143 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:17:02.530109    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:17:24.955749 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:17:24.955714    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-1e1abd/auth-test-model-1e1abd-kserve-58887c88c-llbbh"]
Jul 23 12:17:24.959081 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:17:24.959064    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-1e1abd/auth-test-model-1e1abd-kserve-58887c88c-llbbh"
Jul 23 12:17:24.961980 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:17:24.961959    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-1e1abd\"/\"openshift-service-ca.crt\""
Jul 23 12:17:24.962905 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:17:24.962882    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-1e1abd\"/\"auth-test-model-1e1abd-kserve-self-signed-certs\""
Jul 23 12:17:24.963013 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:17:24.962977    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-1e1abd\"/\"default-dockercfg-fr5ck\""
Jul 23 12:17:24.963057 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:17:24.963037    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-1e1abd\"/\"kube-root-ca.crt\""
Jul 23 12:17:24.974633 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:17:24.974612    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-1e1abd/auth-test-model-1e1abd-kserve-58887c88c-llbbh"]
Jul 23 12:17:24.986289 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:17:24.986229    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:17:25.102216 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:17:25.102161    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e6de6bbc-99b8-4a2d-ab4f-98514c414417-home\") pod \"auth-test-model-1e1abd-kserve-58887c88c-llbbh\" (UID: \"e6de6bbc-99b8-4a2d-ab4f-98514c414417\") " pod="ai-tenant-e2e-shared-b-1e1abd/auth-test-model-1e1abd-kserve-58887c88c-llbbh"
Jul 23 12:17:25.102422 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:17:25.102327    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqk6t\" (UniqueName: \"kubernetes.io/projected/e6de6bbc-99b8-4a2d-ab4f-98514c414417-kube-api-access-wqk6t\") pod \"auth-test-model-1e1abd-kserve-58887c88c-llbbh\" (UID: \"e6de6bbc-99b8-4a2d-ab4f-98514c414417\") " pod="ai-tenant-e2e-shared-b-1e1abd/auth-test-model-1e1abd-kserve-58887c88c-llbbh"
Jul 23 12:17:25.102422 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:17:25.102356    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e6de6bbc-99b8-4a2d-ab4f-98514c414417-dshm\") pod \"auth-test-model-1e1abd-kserve-58887c88c-llbbh\" (UID: \"e6de6bbc-99b8-4a2d-ab4f-98514c414417\") " pod="ai-tenant-e2e-shared-b-1e1abd/auth-test-model-1e1abd-kserve-58887c88c-llbbh"
Jul 23 12:17:25.102529 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:17:25.102420    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6de6bbc-99b8-4a2d-ab4f-98514c414417-model-cache\") pod \"auth-test-model-1e1abd-kserve-58887c88c-llbbh\" (UID: \"e6de6bbc-99b8-4a2d-ab4f-98514c414417\") " pod="ai-tenant-e2e-shared-b-1e1abd/auth-test-model-1e1abd-kserve-58887c88c-llbbh"
Jul 23 12:17:25.102529 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:17:25.102460    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6de6bbc-99b8-4a2d-ab4f-98514c414417-tls-certs\") pod \"auth-test-model-1e1abd-kserve-58887c88c-llbbh\" (UID: \"e6de6bbc-99b8-4a2d-ab4f-98514c414417\") " pod="ai-tenant-e2e-shared-b-1e1abd/auth-test-model-1e1abd-kserve-58887c88c-llbbh"
Jul 23 12:17:25.102529 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:17:25.102516    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6de6bbc-99b8-4a2d-ab4f-98514c414417-tmp-dir\") pod \"auth-test-model-1e1abd-kserve-58887c88c-llbbh\" (UID: \"e6de6bbc-99b8-4a2d-ab4f-98514c414417\") " pod="ai-tenant-e2e-shared-b-1e1abd/auth-test-model-1e1abd-kserve-58887c88c-llbbh"
Jul 23 12:17:25.203491 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:17:25.203453    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqk6t\" (UniqueName: \"kubernetes.io/projected/e6de6bbc-99b8-4a2d-ab4f-98514c414417-kube-api-access-wqk6t\") pod \"auth-test-model-1e1abd-kserve-58887c88c-llbbh\" (UID: \"e6de6bbc-99b8-4a2d-ab4f-98514c414417\") " pod="ai-tenant-e2e-shared-b-1e1abd/auth-test-model-1e1abd-kserve-58887c88c-llbbh"
Jul 23 12:17:25.203491 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:17:25.203490    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e6de6bbc-99b8-4a2d-ab4f-98514c414417-dshm\") pod \"auth-test-model-1e1abd-kserve-58887c88c-llbbh\" (UID: \"e6de6bbc-99b8-4a2d-ab4f-98514c414417\") " pod="ai-tenant-e2e-shared-b-1e1abd/auth-test-model-1e1abd-kserve-58887c88c-llbbh"
Jul 23 12:17:25.203810 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:17:25.203519    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6de6bbc-99b8-4a2d-ab4f-98514c414417-model-cache\") pod \"auth-test-model-1e1abd-kserve-58887c88c-llbbh\" (UID: \"e6de6bbc-99b8-4a2d-ab4f-98514c414417\") " pod="ai-tenant-e2e-shared-b-1e1abd/auth-test-model-1e1abd-kserve-58887c88c-llbbh"
Jul 23 12:17:25.203810 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:17:25.203542    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6de6bbc-99b8-4a2d-ab4f-98514c414417-tls-certs\") pod \"auth-test-model-1e1abd-kserve-58887c88c-llbbh\" (UID: \"e6de6bbc-99b8-4a2d-ab4f-98514c414417\") " pod="ai-tenant-e2e-shared-b-1e1abd/auth-test-model-1e1abd-kserve-58887c88c-llbbh"
Jul 23 12:17:25.203810 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:17:25.203583    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6de6bbc-99b8-4a2d-ab4f-98514c414417-tmp-dir\") pod \"auth-test-model-1e1abd-kserve-58887c88c-llbbh\" (UID: \"e6de6bbc-99b8-4a2d-ab4f-98514c414417\") " pod="ai-tenant-e2e-shared-b-1e1abd/auth-test-model-1e1abd-kserve-58887c88c-llbbh"
Jul 23 12:17:25.203810 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:17:25.203647    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e6de6bbc-99b8-4a2d-ab4f-98514c414417-home\") pod \"auth-test-model-1e1abd-kserve-58887c88c-llbbh\" (UID: \"e6de6bbc-99b8-4a2d-ab4f-98514c414417\") " pod="ai-tenant-e2e-shared-b-1e1abd/auth-test-model-1e1abd-kserve-58887c88c-llbbh"
Jul 23 12:17:25.204059 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:17:25.204030    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6de6bbc-99b8-4a2d-ab4f-98514c414417-tmp-dir\") pod \"auth-test-model-1e1abd-kserve-58887c88c-llbbh\" (UID: \"e6de6bbc-99b8-4a2d-ab4f-98514c414417\") " pod="ai-tenant-e2e-shared-b-1e1abd/auth-test-model-1e1abd-kserve-58887c88c-llbbh"
Jul 23 12:17:25.204143 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:17:25.204040    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6de6bbc-99b8-4a2d-ab4f-98514c414417-model-cache\") pod \"auth-test-model-1e1abd-kserve-58887c88c-llbbh\" (UID: \"e6de6bbc-99b8-4a2d-ab4f-98514c414417\") " pod="ai-tenant-e2e-shared-b-1e1abd/auth-test-model-1e1abd-kserve-58887c88c-llbbh"
Jul 23 12:17:25.204143 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:17:25.204075    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e6de6bbc-99b8-4a2d-ab4f-98514c414417-home\") pod \"auth-test-model-1e1abd-kserve-58887c88c-llbbh\" (UID: \"e6de6bbc-99b8-4a2d-ab4f-98514c414417\") " pod="ai-tenant-e2e-shared-b-1e1abd/auth-test-model-1e1abd-kserve-58887c88c-llbbh"
Jul 23 12:17:25.205963 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:17:25.205900    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e6de6bbc-99b8-4a2d-ab4f-98514c414417-dshm\") pod \"auth-test-model-1e1abd-kserve-58887c88c-llbbh\" (UID: \"e6de6bbc-99b8-4a2d-ab4f-98514c414417\") " pod="ai-tenant-e2e-shared-b-1e1abd/auth-test-model-1e1abd-kserve-58887c88c-llbbh"
Jul 23 12:17:25.206174 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:17:25.206158    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6de6bbc-99b8-4a2d-ab4f-98514c414417-tls-certs\") pod \"auth-test-model-1e1abd-kserve-58887c88c-llbbh\" (UID: \"e6de6bbc-99b8-4a2d-ab4f-98514c414417\") " pod="ai-tenant-e2e-shared-b-1e1abd/auth-test-model-1e1abd-kserve-58887c88c-llbbh"
Jul 23 12:17:25.212756 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:17:25.212736    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqk6t\" (UniqueName: \"kubernetes.io/projected/e6de6bbc-99b8-4a2d-ab4f-98514c414417-kube-api-access-wqk6t\") pod \"auth-test-model-1e1abd-kserve-58887c88c-llbbh\" (UID: \"e6de6bbc-99b8-4a2d-ab4f-98514c414417\") " pod="ai-tenant-e2e-shared-b-1e1abd/auth-test-model-1e1abd-kserve-58887c88c-llbbh"
Jul 23 12:17:25.268735 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:17:25.268675    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-1e1abd/auth-test-model-1e1abd-kserve-58887c88c-llbbh"
Jul 23 12:17:25.402152 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:17:25.402127    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-1e1abd/auth-test-model-1e1abd-kserve-58887c88c-llbbh"]
Jul 23 12:17:25.404244 ip-10-0-141-229 kubenswrapper[2563]: W0723 12:17:25.404213    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode6de6bbc_99b8_4a2d_ab4f_98514c414417.slice/crio-50d48cc85bd8721c485944a6d00a96219478cb1568141f51a56120c62c97dc9a WatchSource:0}: Error finding container 50d48cc85bd8721c485944a6d00a96219478cb1568141f51a56120c62c97dc9a: Status 404 returned error can't find the container with id 50d48cc85bd8721c485944a6d00a96219478cb1568141f51a56120c62c97dc9a
Jul 23 12:17:25.835671 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:17:25.835560    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-1e1abd/auth-test-model-1e1abd-kserve-58887c88c-llbbh" event={"ID":"e6de6bbc-99b8-4a2d-ab4f-98514c414417","Type":"ContainerStarted","Data":"1efa551a126d14610590d9b1c3f701229fe180129471e787ba2175d24c62857b"}
Jul 23 12:17:25.835671 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:17:25.835618    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-1e1abd/auth-test-model-1e1abd-kserve-58887c88c-llbbh" event={"ID":"e6de6bbc-99b8-4a2d-ab4f-98514c414417","Type":"ContainerStarted","Data":"50d48cc85bd8721c485944a6d00a96219478cb1568141f51a56120c62c97dc9a"}
Jul 23 12:17:25.855050 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:17:25.854996    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-1e1abd/auth-test-model-1e1abd-kserve-58887c88c-llbbh" podStartSLOduration=1.854979799 podStartE2EDuration="1.854979799s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 12:17:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 12:17:25.854548088 +0000 UTC m=+2500.247485876" watchObservedRunningTime="2026-07-23 12:17:25.854979799 +0000 UTC m=+2500.247917586"
Jul 23 12:17:35.269469 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:17:35.269426    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-1e1abd/auth-test-model-1e1abd-kserve-58887c88c-llbbh"
Jul 23 12:17:35.269469 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:17:35.269480    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-1e1abd/auth-test-model-1e1abd-kserve-58887c88c-llbbh"
Jul 23 12:17:35.281391 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:17:35.281370    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-1e1abd/auth-test-model-1e1abd-kserve-58887c88c-llbbh"
Jul 23 12:17:35.892990 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:17:35.892961    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-1e1abd/auth-test-model-1e1abd-kserve-58887c88c-llbbh"
Jul 23 12:19:37.606967 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:19:37.606932    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-290648/auth-test-model-290648-kserve-6c5fc6967-hg22n"]
Jul 23 12:19:37.656248 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:19:37.656205    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:19:38.563177 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:19:38.563138    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-1e1abd/auth-test-model-1e1abd-kserve-58887c88c-llbbh"]
Jul 23 12:19:39.603631 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:19:39.603594    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:19:48.287515 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:19:48.287466    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:19:52.626779 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:19:52.626738    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-290648/auth-test-model-290648-kserve-6c5fc6967-hg22n" podUID="bb0aa79d-41a3-49d7-95a7-4c4a96e2d89a" containerName="main" containerID="cri-o://dcfc8482d1c505698138d05a746ebc40172040d8ce8f69b3a87fe0cf8391b703" gracePeriod=45
Jul 23 12:19:52.875230 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:19:52.875205    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-290648/auth-test-model-290648-kserve-6c5fc6967-hg22n"
Jul 23 12:19:52.972932 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:19:52.972847    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb0aa79d-41a3-49d7-95a7-4c4a96e2d89a-tmp-dir\") pod \"bb0aa79d-41a3-49d7-95a7-4c4a96e2d89a\" (UID: \"bb0aa79d-41a3-49d7-95a7-4c4a96e2d89a\") "
Jul 23 12:19:52.972932 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:19:52.972891    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb0aa79d-41a3-49d7-95a7-4c4a96e2d89a-model-cache\") pod \"bb0aa79d-41a3-49d7-95a7-4c4a96e2d89a\" (UID: \"bb0aa79d-41a3-49d7-95a7-4c4a96e2d89a\") "
Jul 23 12:19:52.972932 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:19:52.972911    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb0aa79d-41a3-49d7-95a7-4c4a96e2d89a-dshm\") pod \"bb0aa79d-41a3-49d7-95a7-4c4a96e2d89a\" (UID: \"bb0aa79d-41a3-49d7-95a7-4c4a96e2d89a\") "
Jul 23 12:19:52.973238 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:19:52.972980    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dsb6v\" (UniqueName: \"kubernetes.io/projected/bb0aa79d-41a3-49d7-95a7-4c4a96e2d89a-kube-api-access-dsb6v\") pod \"bb0aa79d-41a3-49d7-95a7-4c4a96e2d89a\" (UID: \"bb0aa79d-41a3-49d7-95a7-4c4a96e2d89a\") "
Jul 23 12:19:52.973238 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:19:52.973012    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb0aa79d-41a3-49d7-95a7-4c4a96e2d89a-tls-certs\") pod \"bb0aa79d-41a3-49d7-95a7-4c4a96e2d89a\" (UID: \"bb0aa79d-41a3-49d7-95a7-4c4a96e2d89a\") "
Jul 23 12:19:52.973238 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:19:52.973174    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb0aa79d-41a3-49d7-95a7-4c4a96e2d89a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "bb0aa79d-41a3-49d7-95a7-4c4a96e2d89a" (UID: "bb0aa79d-41a3-49d7-95a7-4c4a96e2d89a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 12:19:52.973238 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:19:52.973195    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bb0aa79d-41a3-49d7-95a7-4c4a96e2d89a-home\") pod \"bb0aa79d-41a3-49d7-95a7-4c4a96e2d89a\" (UID: \"bb0aa79d-41a3-49d7-95a7-4c4a96e2d89a\") "
Jul 23 12:19:52.973238 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:19:52.973207    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb0aa79d-41a3-49d7-95a7-4c4a96e2d89a-model-cache" (OuterVolumeSpecName: "model-cache") pod "bb0aa79d-41a3-49d7-95a7-4c4a96e2d89a" (UID: "bb0aa79d-41a3-49d7-95a7-4c4a96e2d89a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 12:19:52.973494 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:19:52.973398    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb0aa79d-41a3-49d7-95a7-4c4a96e2d89a-home" (OuterVolumeSpecName: "home") pod "bb0aa79d-41a3-49d7-95a7-4c4a96e2d89a" (UID: "bb0aa79d-41a3-49d7-95a7-4c4a96e2d89a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 12:19:52.973566 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:19:52.973547    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb0aa79d-41a3-49d7-95a7-4c4a96e2d89a-tmp-dir\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 23 12:19:52.973614 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:19:52.973572    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb0aa79d-41a3-49d7-95a7-4c4a96e2d89a-model-cache\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 23 12:19:52.973614 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:19:52.973586    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bb0aa79d-41a3-49d7-95a7-4c4a96e2d89a-home\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 23 12:19:52.975291 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:19:52.975271    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb0aa79d-41a3-49d7-95a7-4c4a96e2d89a-dshm" (OuterVolumeSpecName: "dshm") pod "bb0aa79d-41a3-49d7-95a7-4c4a96e2d89a" (UID: "bb0aa79d-41a3-49d7-95a7-4c4a96e2d89a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 12:19:52.975372 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:19:52.975324    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb0aa79d-41a3-49d7-95a7-4c4a96e2d89a-kube-api-access-dsb6v" (OuterVolumeSpecName: "kube-api-access-dsb6v") pod "bb0aa79d-41a3-49d7-95a7-4c4a96e2d89a" (UID: "bb0aa79d-41a3-49d7-95a7-4c4a96e2d89a"). InnerVolumeSpecName "kube-api-access-dsb6v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 12:19:52.975410 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:19:52.975377    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb0aa79d-41a3-49d7-95a7-4c4a96e2d89a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "bb0aa79d-41a3-49d7-95a7-4c4a96e2d89a" (UID: "bb0aa79d-41a3-49d7-95a7-4c4a96e2d89a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 12:19:53.074842 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:19:53.074806    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb0aa79d-41a3-49d7-95a7-4c4a96e2d89a-dshm\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 23 12:19:53.074842 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:19:53.074836    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dsb6v\" (UniqueName: \"kubernetes.io/projected/bb0aa79d-41a3-49d7-95a7-4c4a96e2d89a-kube-api-access-dsb6v\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 23 12:19:53.074842 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:19:53.074846    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb0aa79d-41a3-49d7-95a7-4c4a96e2d89a-tls-certs\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 23 12:19:53.528367 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:19:53.528331    2563 generic.go:356] "Generic (PLEG): container finished" podID="bb0aa79d-41a3-49d7-95a7-4c4a96e2d89a" containerID="dcfc8482d1c505698138d05a746ebc40172040d8ce8f69b3a87fe0cf8391b703" exitCode=0
Jul 23 12:19:53.528367 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:19:53.528369    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-290648/auth-test-model-290648-kserve-6c5fc6967-hg22n"
Jul 23 12:19:53.528613 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:19:53.528417    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-290648/auth-test-model-290648-kserve-6c5fc6967-hg22n" event={"ID":"bb0aa79d-41a3-49d7-95a7-4c4a96e2d89a","Type":"ContainerDied","Data":"dcfc8482d1c505698138d05a746ebc40172040d8ce8f69b3a87fe0cf8391b703"}
Jul 23 12:19:53.528613 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:19:53.528456    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-290648/auth-test-model-290648-kserve-6c5fc6967-hg22n" event={"ID":"bb0aa79d-41a3-49d7-95a7-4c4a96e2d89a","Type":"ContainerDied","Data":"c48565de2d365522e00f1231c60e7c961e01bf5b5c2bde714b34b0e578cfdabd"}
Jul 23 12:19:53.528613 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:19:53.528477    2563 scope.go:117] "RemoveContainer" containerID="dcfc8482d1c505698138d05a746ebc40172040d8ce8f69b3a87fe0cf8391b703"
Jul 23 12:19:53.537205 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:19:53.537186    2563 scope.go:117] "RemoveContainer" containerID="dcfc8482d1c505698138d05a746ebc40172040d8ce8f69b3a87fe0cf8391b703"
Jul 23 12:19:53.537440 ip-10-0-141-229 kubenswrapper[2563]: E0723 12:19:53.537417    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dcfc8482d1c505698138d05a746ebc40172040d8ce8f69b3a87fe0cf8391b703\": container with ID starting with dcfc8482d1c505698138d05a746ebc40172040d8ce8f69b3a87fe0cf8391b703 not found: ID does not exist" containerID="dcfc8482d1c505698138d05a746ebc40172040d8ce8f69b3a87fe0cf8391b703"
Jul 23 12:19:53.537489 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:19:53.537450    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dcfc8482d1c505698138d05a746ebc40172040d8ce8f69b3a87fe0cf8391b703"} err="failed to get container status \"dcfc8482d1c505698138d05a746ebc40172040d8ce8f69b3a87fe0cf8391b703\": rpc error: code = NotFound desc = could not find container \"dcfc8482d1c505698138d05a746ebc40172040d8ce8f69b3a87fe0cf8391b703\": container with ID starting with dcfc8482d1c505698138d05a746ebc40172040d8ce8f69b3a87fe0cf8391b703 not found: ID does not exist"
Jul 23 12:19:53.548196 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:19:53.548173    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-290648/auth-test-model-290648-kserve-6c5fc6967-hg22n"]
Jul 23 12:19:53.550852 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:19:53.550832    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-290648/auth-test-model-290648-kserve-6c5fc6967-hg22n"]
Jul 23 12:19:53.583475 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:19:53.583424    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-1e1abd/auth-test-model-1e1abd-kserve-58887c88c-llbbh" podUID="e6de6bbc-99b8-4a2d-ab4f-98514c414417" containerName="main" containerID="cri-o://1efa551a126d14610590d9b1c3f701229fe180129471e787ba2175d24c62857b" gracePeriod=45
Jul 23 12:19:53.846965 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:19:53.846931    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-1e1abd/auth-test-model-1e1abd-kserve-58887c88c-llbbh"
Jul 23 12:19:53.983488 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:19:53.983458    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e6de6bbc-99b8-4a2d-ab4f-98514c414417-home\") pod \"e6de6bbc-99b8-4a2d-ab4f-98514c414417\" (UID: \"e6de6bbc-99b8-4a2d-ab4f-98514c414417\") "
Jul 23 12:19:53.983659 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:19:53.983499    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6de6bbc-99b8-4a2d-ab4f-98514c414417-tls-certs\") pod \"e6de6bbc-99b8-4a2d-ab4f-98514c414417\" (UID: \"e6de6bbc-99b8-4a2d-ab4f-98514c414417\") "
Jul 23 12:19:53.983659 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:19:53.983518    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e6de6bbc-99b8-4a2d-ab4f-98514c414417-dshm\") pod \"e6de6bbc-99b8-4a2d-ab4f-98514c414417\" (UID: \"e6de6bbc-99b8-4a2d-ab4f-98514c414417\") "
Jul 23 12:19:53.983659 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:19:53.983561    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6de6bbc-99b8-4a2d-ab4f-98514c414417-model-cache\") pod \"e6de6bbc-99b8-4a2d-ab4f-98514c414417\" (UID: \"e6de6bbc-99b8-4a2d-ab4f-98514c414417\") "
Jul 23 12:19:53.983659 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:19:53.983587    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6de6bbc-99b8-4a2d-ab4f-98514c414417-tmp-dir\") pod \"e6de6bbc-99b8-4a2d-ab4f-98514c414417\" (UID: \"e6de6bbc-99b8-4a2d-ab4f-98514c414417\") "
Jul 23 12:19:53.983894 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:19:53.983728    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wqk6t\" (UniqueName: \"kubernetes.io/projected/e6de6bbc-99b8-4a2d-ab4f-98514c414417-kube-api-access-wqk6t\") pod \"e6de6bbc-99b8-4a2d-ab4f-98514c414417\" (UID: \"e6de6bbc-99b8-4a2d-ab4f-98514c414417\") "
Jul 23 12:19:53.983894 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:19:53.983788    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6de6bbc-99b8-4a2d-ab4f-98514c414417-home" (OuterVolumeSpecName: "home") pod "e6de6bbc-99b8-4a2d-ab4f-98514c414417" (UID: "e6de6bbc-99b8-4a2d-ab4f-98514c414417"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 12:19:53.983894 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:19:53.983837    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6de6bbc-99b8-4a2d-ab4f-98514c414417-model-cache" (OuterVolumeSpecName: "model-cache") pod "e6de6bbc-99b8-4a2d-ab4f-98514c414417" (UID: "e6de6bbc-99b8-4a2d-ab4f-98514c414417"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 12:19:53.984039 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:19:53.983930    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6de6bbc-99b8-4a2d-ab4f-98514c414417-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e6de6bbc-99b8-4a2d-ab4f-98514c414417" (UID: "e6de6bbc-99b8-4a2d-ab4f-98514c414417"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 12:19:53.984106 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:19:53.984087    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e6de6bbc-99b8-4a2d-ab4f-98514c414417-home\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 23 12:19:53.984158 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:19:53.984111    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6de6bbc-99b8-4a2d-ab4f-98514c414417-model-cache\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 23 12:19:53.984158 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:19:53.984126    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6de6bbc-99b8-4a2d-ab4f-98514c414417-tmp-dir\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 23 12:19:53.985943 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:19:53.985922    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6de6bbc-99b8-4a2d-ab4f-98514c414417-dshm" (OuterVolumeSpecName: "dshm") pod "e6de6bbc-99b8-4a2d-ab4f-98514c414417" (UID: "e6de6bbc-99b8-4a2d-ab4f-98514c414417"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 12:19:53.985943 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:19:53.985935    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6de6bbc-99b8-4a2d-ab4f-98514c414417-kube-api-access-wqk6t" (OuterVolumeSpecName: "kube-api-access-wqk6t") pod "e6de6bbc-99b8-4a2d-ab4f-98514c414417" (UID: "e6de6bbc-99b8-4a2d-ab4f-98514c414417"). InnerVolumeSpecName "kube-api-access-wqk6t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 12:19:53.986045 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:19:53.985980    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e6de6bbc-99b8-4a2d-ab4f-98514c414417-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e6de6bbc-99b8-4a2d-ab4f-98514c414417" (UID: "e6de6bbc-99b8-4a2d-ab4f-98514c414417"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 12:19:54.078306 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:19:54.078226    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bb0aa79d-41a3-49d7-95a7-4c4a96e2d89a" path="/var/lib/kubelet/pods/bb0aa79d-41a3-49d7-95a7-4c4a96e2d89a/volumes"
Jul 23 12:19:54.085435 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:19:54.085244    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wqk6t\" (UniqueName: \"kubernetes.io/projected/e6de6bbc-99b8-4a2d-ab4f-98514c414417-kube-api-access-wqk6t\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 23 12:19:54.085435 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:19:54.085268    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6de6bbc-99b8-4a2d-ab4f-98514c414417-tls-certs\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 23 12:19:54.085435 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:19:54.085280    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e6de6bbc-99b8-4a2d-ab4f-98514c414417-dshm\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 23 12:19:54.534894 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:19:54.534856    2563 generic.go:356] "Generic (PLEG): container finished" podID="e6de6bbc-99b8-4a2d-ab4f-98514c414417" containerID="1efa551a126d14610590d9b1c3f701229fe180129471e787ba2175d24c62857b" exitCode=0
Jul 23 12:19:54.535079 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:19:54.534901    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-1e1abd/auth-test-model-1e1abd-kserve-58887c88c-llbbh"
Jul 23 12:19:54.535079 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:19:54.534940    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-1e1abd/auth-test-model-1e1abd-kserve-58887c88c-llbbh" event={"ID":"e6de6bbc-99b8-4a2d-ab4f-98514c414417","Type":"ContainerDied","Data":"1efa551a126d14610590d9b1c3f701229fe180129471e787ba2175d24c62857b"}
Jul 23 12:19:54.535079 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:19:54.534981    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-1e1abd/auth-test-model-1e1abd-kserve-58887c88c-llbbh" event={"ID":"e6de6bbc-99b8-4a2d-ab4f-98514c414417","Type":"ContainerDied","Data":"50d48cc85bd8721c485944a6d00a96219478cb1568141f51a56120c62c97dc9a"}
Jul 23 12:19:54.535079 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:19:54.535006    2563 scope.go:117] "RemoveContainer" containerID="1efa551a126d14610590d9b1c3f701229fe180129471e787ba2175d24c62857b"
Jul 23 12:19:54.543398 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:19:54.543379    2563 scope.go:117] "RemoveContainer" containerID="1efa551a126d14610590d9b1c3f701229fe180129471e787ba2175d24c62857b"
Jul 23 12:19:54.543644 ip-10-0-141-229 kubenswrapper[2563]: E0723 12:19:54.543628    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1efa551a126d14610590d9b1c3f701229fe180129471e787ba2175d24c62857b\": container with ID starting with 1efa551a126d14610590d9b1c3f701229fe180129471e787ba2175d24c62857b not found: ID does not exist" containerID="1efa551a126d14610590d9b1c3f701229fe180129471e787ba2175d24c62857b"
Jul 23 12:19:54.543734 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:19:54.543650    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1efa551a126d14610590d9b1c3f701229fe180129471e787ba2175d24c62857b"} err="failed to get container status \"1efa551a126d14610590d9b1c3f701229fe180129471e787ba2175d24c62857b\": rpc error: code = NotFound desc = could not find container \"1efa551a126d14610590d9b1c3f701229fe180129471e787ba2175d24c62857b\": container with ID starting with 1efa551a126d14610590d9b1c3f701229fe180129471e787ba2175d24c62857b not found: ID does not exist"
Jul 23 12:19:54.550503 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:19:54.550476    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-1e1abd/auth-test-model-1e1abd-kserve-58887c88c-llbbh"]
Jul 23 12:19:54.554822 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:19:54.554800    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-1e1abd/auth-test-model-1e1abd-kserve-58887c88c-llbbh"]
Jul 23 12:19:56.078090 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:19:56.078058    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e6de6bbc-99b8-4a2d-ab4f-98514c414417" path="/var/lib/kubelet/pods/e6de6bbc-99b8-4a2d-ab4f-98514c414417/volumes"
Jul 23 12:20:06.349556 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:20:06.349513    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-290648/sub-test-model-290648-kserve-b6d574d89-crxpl"]
Jul 23 12:20:06.349944 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:20:06.349905    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e6de6bbc-99b8-4a2d-ab4f-98514c414417" containerName="main"
Jul 23 12:20:06.349944 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:20:06.349917    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="e6de6bbc-99b8-4a2d-ab4f-98514c414417" containerName="main"
Jul 23 12:20:06.349944 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:20:06.349929    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb0aa79d-41a3-49d7-95a7-4c4a96e2d89a" containerName="main"
Jul 23 12:20:06.349944 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:20:06.349935    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb0aa79d-41a3-49d7-95a7-4c4a96e2d89a" containerName="main"
Jul 23 12:20:06.350068 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:20:06.349998    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="e6de6bbc-99b8-4a2d-ab4f-98514c414417" containerName="main"
Jul 23 12:20:06.350068 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:20:06.350009    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="bb0aa79d-41a3-49d7-95a7-4c4a96e2d89a" containerName="main"
Jul 23 12:20:06.354301 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:20:06.354282    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-290648/sub-test-model-290648-kserve-b6d574d89-crxpl"
Jul 23 12:20:06.357529 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:20:06.357504    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-290648\"/\"default-dockercfg-4655d\""
Jul 23 12:20:06.357629 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:20:06.357507    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-290648\"/\"sub-test-model-290648-kserve-self-signed-certs\""
Jul 23 12:20:06.357629 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:20:06.357508    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-290648\"/\"kube-root-ca.crt\""
Jul 23 12:20:06.357752 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:20:06.357513    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-290648\"/\"openshift-service-ca.crt\""
Jul 23 12:20:06.363493 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:20:06.363472    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-290648/sub-test-model-290648-kserve-b6d574d89-crxpl"]
Jul 23 12:20:06.375965 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:20:06.375941    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:20:06.494871 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:20:06.494838    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4027c1ac-03a5-4918-b917-2c134741e4b5-dshm\") pod \"sub-test-model-290648-kserve-b6d574d89-crxpl\" (UID: \"4027c1ac-03a5-4918-b917-2c134741e4b5\") " pod="ai-tenant-e2e-shared-a-290648/sub-test-model-290648-kserve-b6d574d89-crxpl"
Jul 23 12:20:06.495053 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:20:06.494911    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4027c1ac-03a5-4918-b917-2c134741e4b5-tls-certs\") pod \"sub-test-model-290648-kserve-b6d574d89-crxpl\" (UID: \"4027c1ac-03a5-4918-b917-2c134741e4b5\") " pod="ai-tenant-e2e-shared-a-290648/sub-test-model-290648-kserve-b6d574d89-crxpl"
Jul 23 12:20:06.495053 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:20:06.494962    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4027c1ac-03a5-4918-b917-2c134741e4b5-model-cache\") pod \"sub-test-model-290648-kserve-b6d574d89-crxpl\" (UID: \"4027c1ac-03a5-4918-b917-2c134741e4b5\") " pod="ai-tenant-e2e-shared-a-290648/sub-test-model-290648-kserve-b6d574d89-crxpl"
Jul 23 12:20:06.495053 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:20:06.494992    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2xbc\" (UniqueName: \"kubernetes.io/projected/4027c1ac-03a5-4918-b917-2c134741e4b5-kube-api-access-d2xbc\") pod \"sub-test-model-290648-kserve-b6d574d89-crxpl\" (UID: \"4027c1ac-03a5-4918-b917-2c134741e4b5\") " pod="ai-tenant-e2e-shared-a-290648/sub-test-model-290648-kserve-b6d574d89-crxpl"
Jul 23 12:20:06.495053 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:20:06.495044    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4027c1ac-03a5-4918-b917-2c134741e4b5-home\") pod \"sub-test-model-290648-kserve-b6d574d89-crxpl\" (UID: \"4027c1ac-03a5-4918-b917-2c134741e4b5\") " pod="ai-tenant-e2e-shared-a-290648/sub-test-model-290648-kserve-b6d574d89-crxpl"
Jul 23 12:20:06.495203 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:20:06.495059    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4027c1ac-03a5-4918-b917-2c134741e4b5-tmp-dir\") pod \"sub-test-model-290648-kserve-b6d574d89-crxpl\" (UID: \"4027c1ac-03a5-4918-b917-2c134741e4b5\") " pod="ai-tenant-e2e-shared-a-290648/sub-test-model-290648-kserve-b6d574d89-crxpl"
Jul 23 12:20:06.596094 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:20:06.596059    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4027c1ac-03a5-4918-b917-2c134741e4b5-home\") pod \"sub-test-model-290648-kserve-b6d574d89-crxpl\" (UID: \"4027c1ac-03a5-4918-b917-2c134741e4b5\") " pod="ai-tenant-e2e-shared-a-290648/sub-test-model-290648-kserve-b6d574d89-crxpl"
Jul 23 12:20:06.596297 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:20:06.596100    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4027c1ac-03a5-4918-b917-2c134741e4b5-tmp-dir\") pod \"sub-test-model-290648-kserve-b6d574d89-crxpl\" (UID: \"4027c1ac-03a5-4918-b917-2c134741e4b5\") " pod="ai-tenant-e2e-shared-a-290648/sub-test-model-290648-kserve-b6d574d89-crxpl"
Jul 23 12:20:06.596297 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:20:06.596143    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4027c1ac-03a5-4918-b917-2c134741e4b5-dshm\") pod \"sub-test-model-290648-kserve-b6d574d89-crxpl\" (UID: \"4027c1ac-03a5-4918-b917-2c134741e4b5\") " pod="ai-tenant-e2e-shared-a-290648/sub-test-model-290648-kserve-b6d574d89-crxpl"
Jul 23 12:20:06.596297 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:20:06.596189    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4027c1ac-03a5-4918-b917-2c134741e4b5-tls-certs\") pod \"sub-test-model-290648-kserve-b6d574d89-crxpl\" (UID: \"4027c1ac-03a5-4918-b917-2c134741e4b5\") " pod="ai-tenant-e2e-shared-a-290648/sub-test-model-290648-kserve-b6d574d89-crxpl"
Jul 23 12:20:06.596297 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:20:06.596206    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4027c1ac-03a5-4918-b917-2c134741e4b5-model-cache\") pod \"sub-test-model-290648-kserve-b6d574d89-crxpl\" (UID: \"4027c1ac-03a5-4918-b917-2c134741e4b5\") " pod="ai-tenant-e2e-shared-a-290648/sub-test-model-290648-kserve-b6d574d89-crxpl"
Jul 23 12:20:06.596297 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:20:06.596226    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2xbc\" (UniqueName: \"kubernetes.io/projected/4027c1ac-03a5-4918-b917-2c134741e4b5-kube-api-access-d2xbc\") pod \"sub-test-model-290648-kserve-b6d574d89-crxpl\" (UID: \"4027c1ac-03a5-4918-b917-2c134741e4b5\") " pod="ai-tenant-e2e-shared-a-290648/sub-test-model-290648-kserve-b6d574d89-crxpl"
Jul 23 12:20:06.596628 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:20:06.596501    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4027c1ac-03a5-4918-b917-2c134741e4b5-home\") pod \"sub-test-model-290648-kserve-b6d574d89-crxpl\" (UID: \"4027c1ac-03a5-4918-b917-2c134741e4b5\") " pod="ai-tenant-e2e-shared-a-290648/sub-test-model-290648-kserve-b6d574d89-crxpl"
Jul 23 12:20:06.596628 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:20:06.596560    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4027c1ac-03a5-4918-b917-2c134741e4b5-tmp-dir\") pod \"sub-test-model-290648-kserve-b6d574d89-crxpl\" (UID: \"4027c1ac-03a5-4918-b917-2c134741e4b5\") " pod="ai-tenant-e2e-shared-a-290648/sub-test-model-290648-kserve-b6d574d89-crxpl"
Jul 23 12:20:06.596628 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:20:06.596570    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4027c1ac-03a5-4918-b917-2c134741e4b5-model-cache\") pod \"sub-test-model-290648-kserve-b6d574d89-crxpl\" (UID: \"4027c1ac-03a5-4918-b917-2c134741e4b5\") " pod="ai-tenant-e2e-shared-a-290648/sub-test-model-290648-kserve-b6d574d89-crxpl"
Jul 23 12:20:06.598719 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:20:06.598678    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4027c1ac-03a5-4918-b917-2c134741e4b5-dshm\") pod \"sub-test-model-290648-kserve-b6d574d89-crxpl\" (UID: \"4027c1ac-03a5-4918-b917-2c134741e4b5\") " pod="ai-tenant-e2e-shared-a-290648/sub-test-model-290648-kserve-b6d574d89-crxpl"
Jul 23 12:20:06.598907 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:20:06.598888    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4027c1ac-03a5-4918-b917-2c134741e4b5-tls-certs\") pod \"sub-test-model-290648-kserve-b6d574d89-crxpl\" (UID: \"4027c1ac-03a5-4918-b917-2c134741e4b5\") " pod="ai-tenant-e2e-shared-a-290648/sub-test-model-290648-kserve-b6d574d89-crxpl"
Jul 23 12:20:06.604704 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:20:06.604638    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2xbc\" (UniqueName: \"kubernetes.io/projected/4027c1ac-03a5-4918-b917-2c134741e4b5-kube-api-access-d2xbc\") pod \"sub-test-model-290648-kserve-b6d574d89-crxpl\" (UID: \"4027c1ac-03a5-4918-b917-2c134741e4b5\") " pod="ai-tenant-e2e-shared-a-290648/sub-test-model-290648-kserve-b6d574d89-crxpl"
Jul 23 12:20:06.664753 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:20:06.664715    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-290648/sub-test-model-290648-kserve-b6d574d89-crxpl"
Jul 23 12:20:06.800171 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:20:06.800139    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-290648/sub-test-model-290648-kserve-b6d574d89-crxpl"]
Jul 23 12:20:06.803762 ip-10-0-141-229 kubenswrapper[2563]: W0723 12:20:06.803730    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4027c1ac_03a5_4918_b917_2c134741e4b5.slice/crio-36b1cb0bb66282b66282fedfef6921908ff130886cb92ed27d2a513f01bd130c WatchSource:0}: Error finding container 36b1cb0bb66282b66282fedfef6921908ff130886cb92ed27d2a513f01bd130c: Status 404 returned error can't find the container with id 36b1cb0bb66282b66282fedfef6921908ff130886cb92ed27d2a513f01bd130c
Jul 23 12:20:07.596136 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:20:07.596098    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-290648/sub-test-model-290648-kserve-b6d574d89-crxpl" event={"ID":"4027c1ac-03a5-4918-b917-2c134741e4b5","Type":"ContainerStarted","Data":"bd7fce4ea6ad43eae7f752d5bdb2ce2ae0ff5238dab75d46bd8b618d0426285f"}
Jul 23 12:20:07.596136 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:20:07.596140    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-290648/sub-test-model-290648-kserve-b6d574d89-crxpl" event={"ID":"4027c1ac-03a5-4918-b917-2c134741e4b5","Type":"ContainerStarted","Data":"36b1cb0bb66282b66282fedfef6921908ff130886cb92ed27d2a513f01bd130c"}
Jul 23 12:20:07.613828 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:20:07.613770    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-290648/sub-test-model-290648-kserve-b6d574d89-crxpl" podStartSLOduration=1.613755021 podStartE2EDuration="1.613755021s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 12:20:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 12:20:07.612409263 +0000 UTC m=+2662.005347051" watchObservedRunningTime="2026-07-23 12:20:07.613755021 +0000 UTC m=+2662.006692809"
Jul 23 12:20:16.665475 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:20:16.665426    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-290648/sub-test-model-290648-kserve-b6d574d89-crxpl"
Jul 23 12:20:16.665475 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:20:16.665488    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-290648/sub-test-model-290648-kserve-b6d574d89-crxpl"
Jul 23 12:20:16.678071 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:20:16.678037    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-290648/sub-test-model-290648-kserve-b6d574d89-crxpl"
Jul 23 12:20:17.653458 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:20:17.653425    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-290648/sub-test-model-290648-kserve-b6d574d89-crxpl"
Jul 23 12:20:34.280948 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:20:34.280904    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:20:46.212720 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:20:46.212621    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nhm9p_93649ac9-9062-46c0-adaa-1048d471211a/ovn-acl-logging/0.log"
Jul 23 12:20:46.218373 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:20:46.218350    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nhm9p_93649ac9-9062-46c0-adaa-1048d471211a/ovn-acl-logging/0.log"
Jul 23 12:20:53.720954 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:20:53.720909    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:20:53.885013 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:20:53.884977    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:21:31.357324 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:21:31.357286    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-290648/sub-test-model-290648-kserve-b6d574d89-crxpl"]
Jul 23 12:21:31.376042 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:21:31.376014    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:21:32.361075 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:21:32.361038    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:21:40.239009 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:21:40.238976    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-290648/rate-test-model-290648-kserve-5675db7d67-sx7bd"]
Jul 23 12:21:40.242180 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:21:40.242161    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-290648/rate-test-model-290648-kserve-5675db7d67-sx7bd"
Jul 23 12:21:40.244379 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:21:40.244361    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-290648\"/\"rate-test-model-290648-kserve-self-signed-certs\""
Jul 23 12:21:40.252363 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:21:40.252343    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-290648/rate-test-model-290648-kserve-5675db7d67-sx7bd"]
Jul 23 12:21:40.266036 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:21:40.266008    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:21:40.357207 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:21:40.357169    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e565cc2d-74a0-47fe-be0c-d669f308e3ce-tmp-dir\") pod \"rate-test-model-290648-kserve-5675db7d67-sx7bd\" (UID: \"e565cc2d-74a0-47fe-be0c-d669f308e3ce\") " pod="ai-tenant-e2e-shared-a-290648/rate-test-model-290648-kserve-5675db7d67-sx7bd"
Jul 23 12:21:40.357390 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:21:40.357215    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e565cc2d-74a0-47fe-be0c-d669f308e3ce-home\") pod \"rate-test-model-290648-kserve-5675db7d67-sx7bd\" (UID: \"e565cc2d-74a0-47fe-be0c-d669f308e3ce\") " pod="ai-tenant-e2e-shared-a-290648/rate-test-model-290648-kserve-5675db7d67-sx7bd"
Jul 23 12:21:40.357390 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:21:40.357241    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e565cc2d-74a0-47fe-be0c-d669f308e3ce-model-cache\") pod \"rate-test-model-290648-kserve-5675db7d67-sx7bd\" (UID: \"e565cc2d-74a0-47fe-be0c-d669f308e3ce\") " pod="ai-tenant-e2e-shared-a-290648/rate-test-model-290648-kserve-5675db7d67-sx7bd"
Jul 23 12:21:40.357390 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:21:40.357321    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e565cc2d-74a0-47fe-be0c-d669f308e3ce-tls-certs\") pod \"rate-test-model-290648-kserve-5675db7d67-sx7bd\" (UID: \"e565cc2d-74a0-47fe-be0c-d669f308e3ce\") " pod="ai-tenant-e2e-shared-a-290648/rate-test-model-290648-kserve-5675db7d67-sx7bd"
Jul 23 12:21:40.357390 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:21:40.357350    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e565cc2d-74a0-47fe-be0c-d669f308e3ce-dshm\") pod \"rate-test-model-290648-kserve-5675db7d67-sx7bd\" (UID: \"e565cc2d-74a0-47fe-be0c-d669f308e3ce\") " pod="ai-tenant-e2e-shared-a-290648/rate-test-model-290648-kserve-5675db7d67-sx7bd"
Jul 23 12:21:40.357390 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:21:40.357369    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wlp4w\" (UniqueName: \"kubernetes.io/projected/e565cc2d-74a0-47fe-be0c-d669f308e3ce-kube-api-access-wlp4w\") pod \"rate-test-model-290648-kserve-5675db7d67-sx7bd\" (UID: \"e565cc2d-74a0-47fe-be0c-d669f308e3ce\") " pod="ai-tenant-e2e-shared-a-290648/rate-test-model-290648-kserve-5675db7d67-sx7bd"
Jul 23 12:21:40.458859 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:21:40.458820    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e565cc2d-74a0-47fe-be0c-d669f308e3ce-tls-certs\") pod \"rate-test-model-290648-kserve-5675db7d67-sx7bd\" (UID: \"e565cc2d-74a0-47fe-be0c-d669f308e3ce\") " pod="ai-tenant-e2e-shared-a-290648/rate-test-model-290648-kserve-5675db7d67-sx7bd"
Jul 23 12:21:40.459029 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:21:40.458869    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e565cc2d-74a0-47fe-be0c-d669f308e3ce-dshm\") pod \"rate-test-model-290648-kserve-5675db7d67-sx7bd\" (UID: \"e565cc2d-74a0-47fe-be0c-d669f308e3ce\") " pod="ai-tenant-e2e-shared-a-290648/rate-test-model-290648-kserve-5675db7d67-sx7bd"
Jul 23 12:21:40.459029 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:21:40.458902    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wlp4w\" (UniqueName: \"kubernetes.io/projected/e565cc2d-74a0-47fe-be0c-d669f308e3ce-kube-api-access-wlp4w\") pod \"rate-test-model-290648-kserve-5675db7d67-sx7bd\" (UID: \"e565cc2d-74a0-47fe-be0c-d669f308e3ce\") " pod="ai-tenant-e2e-shared-a-290648/rate-test-model-290648-kserve-5675db7d67-sx7bd"
Jul 23 12:21:40.459103 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:21:40.459056    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e565cc2d-74a0-47fe-be0c-d669f308e3ce-tmp-dir\") pod \"rate-test-model-290648-kserve-5675db7d67-sx7bd\" (UID: \"e565cc2d-74a0-47fe-be0c-d669f308e3ce\") " pod="ai-tenant-e2e-shared-a-290648/rate-test-model-290648-kserve-5675db7d67-sx7bd"
Jul 23 12:21:40.459151 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:21:40.459116    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e565cc2d-74a0-47fe-be0c-d669f308e3ce-home\") pod \"rate-test-model-290648-kserve-5675db7d67-sx7bd\" (UID: \"e565cc2d-74a0-47fe-be0c-d669f308e3ce\") " pod="ai-tenant-e2e-shared-a-290648/rate-test-model-290648-kserve-5675db7d67-sx7bd"
Jul 23 12:21:40.459151 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:21:40.459141    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e565cc2d-74a0-47fe-be0c-d669f308e3ce-model-cache\") pod \"rate-test-model-290648-kserve-5675db7d67-sx7bd\" (UID: \"e565cc2d-74a0-47fe-be0c-d669f308e3ce\") " pod="ai-tenant-e2e-shared-a-290648/rate-test-model-290648-kserve-5675db7d67-sx7bd"
Jul 23 12:21:40.459398 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:21:40.459379    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e565cc2d-74a0-47fe-be0c-d669f308e3ce-tmp-dir\") pod \"rate-test-model-290648-kserve-5675db7d67-sx7bd\" (UID: \"e565cc2d-74a0-47fe-be0c-d669f308e3ce\") " pod="ai-tenant-e2e-shared-a-290648/rate-test-model-290648-kserve-5675db7d67-sx7bd"
Jul 23 12:21:40.459477 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:21:40.459456    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e565cc2d-74a0-47fe-be0c-d669f308e3ce-home\") pod \"rate-test-model-290648-kserve-5675db7d67-sx7bd\" (UID: \"e565cc2d-74a0-47fe-be0c-d669f308e3ce\") " pod="ai-tenant-e2e-shared-a-290648/rate-test-model-290648-kserve-5675db7d67-sx7bd"
Jul 23 12:21:40.459539 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:21:40.459461    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e565cc2d-74a0-47fe-be0c-d669f308e3ce-model-cache\") pod \"rate-test-model-290648-kserve-5675db7d67-sx7bd\" (UID: \"e565cc2d-74a0-47fe-be0c-d669f308e3ce\") " pod="ai-tenant-e2e-shared-a-290648/rate-test-model-290648-kserve-5675db7d67-sx7bd"
Jul 23 12:21:40.461324 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:21:40.461298    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e565cc2d-74a0-47fe-be0c-d669f308e3ce-dshm\") pod \"rate-test-model-290648-kserve-5675db7d67-sx7bd\" (UID: \"e565cc2d-74a0-47fe-be0c-d669f308e3ce\") " pod="ai-tenant-e2e-shared-a-290648/rate-test-model-290648-kserve-5675db7d67-sx7bd"
Jul 23 12:21:40.461582 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:21:40.461565    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e565cc2d-74a0-47fe-be0c-d669f308e3ce-tls-certs\") pod \"rate-test-model-290648-kserve-5675db7d67-sx7bd\" (UID: \"e565cc2d-74a0-47fe-be0c-d669f308e3ce\") " pod="ai-tenant-e2e-shared-a-290648/rate-test-model-290648-kserve-5675db7d67-sx7bd"
Jul 23 12:21:40.467141 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:21:40.467123    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wlp4w\" (UniqueName: \"kubernetes.io/projected/e565cc2d-74a0-47fe-be0c-d669f308e3ce-kube-api-access-wlp4w\") pod \"rate-test-model-290648-kserve-5675db7d67-sx7bd\" (UID: \"e565cc2d-74a0-47fe-be0c-d669f308e3ce\") " pod="ai-tenant-e2e-shared-a-290648/rate-test-model-290648-kserve-5675db7d67-sx7bd"
Jul 23 12:21:40.551932 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:21:40.551836    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-290648/rate-test-model-290648-kserve-5675db7d67-sx7bd"
Jul 23 12:21:40.684177 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:21:40.684151    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-290648/rate-test-model-290648-kserve-5675db7d67-sx7bd"]
Jul 23 12:21:40.686236 ip-10-0-141-229 kubenswrapper[2563]: W0723 12:21:40.686203    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode565cc2d_74a0_47fe_be0c_d669f308e3ce.slice/crio-043761b9dd8ab58370f25395c520c36eac5d1eebb6d28c59d8af2474fb431540 WatchSource:0}: Error finding container 043761b9dd8ab58370f25395c520c36eac5d1eebb6d28c59d8af2474fb431540: Status 404 returned error can't find the container with id 043761b9dd8ab58370f25395c520c36eac5d1eebb6d28c59d8af2474fb431540
Jul 23 12:21:41.036476 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:21:41.036436    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-290648/rate-test-model-290648-kserve-5675db7d67-sx7bd" event={"ID":"e565cc2d-74a0-47fe-be0c-d669f308e3ce","Type":"ContainerStarted","Data":"a2db0d2129f782566d312baa7c5dcc26fd675e53e348f7260ff6e009673b21c8"}
Jul 23 12:21:41.036476 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:21:41.036492    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-290648/rate-test-model-290648-kserve-5675db7d67-sx7bd" event={"ID":"e565cc2d-74a0-47fe-be0c-d669f308e3ce","Type":"ContainerStarted","Data":"043761b9dd8ab58370f25395c520c36eac5d1eebb6d28c59d8af2474fb431540"}
Jul 23 12:21:41.053679 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:21:41.053624    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-290648/rate-test-model-290648-kserve-5675db7d67-sx7bd" podStartSLOduration=1.053609693 podStartE2EDuration="1.053609693s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 12:21:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 12:21:41.052135183 +0000 UTC m=+2755.445072971" watchObservedRunningTime="2026-07-23 12:21:41.053609693 +0000 UTC m=+2755.446547479"
Jul 23 12:21:46.377203 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:21:46.377137    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-290648/sub-test-model-290648-kserve-b6d574d89-crxpl" podUID="4027c1ac-03a5-4918-b917-2c134741e4b5" containerName="main" containerID="cri-o://bd7fce4ea6ad43eae7f752d5bdb2ce2ae0ff5238dab75d46bd8b618d0426285f" gracePeriod=45
Jul 23 12:21:46.628168 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:21:46.628107    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-290648/sub-test-model-290648-kserve-b6d574d89-crxpl"
Jul 23 12:21:46.713587 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:21:46.713557    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4027c1ac-03a5-4918-b917-2c134741e4b5-model-cache\") pod \"4027c1ac-03a5-4918-b917-2c134741e4b5\" (UID: \"4027c1ac-03a5-4918-b917-2c134741e4b5\") "
Jul 23 12:21:46.713819 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:21:46.713616    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4027c1ac-03a5-4918-b917-2c134741e4b5-dshm\") pod \"4027c1ac-03a5-4918-b917-2c134741e4b5\" (UID: \"4027c1ac-03a5-4918-b917-2c134741e4b5\") "
Jul 23 12:21:46.713819 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:21:46.713653    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d2xbc\" (UniqueName: \"kubernetes.io/projected/4027c1ac-03a5-4918-b917-2c134741e4b5-kube-api-access-d2xbc\") pod \"4027c1ac-03a5-4918-b917-2c134741e4b5\" (UID: \"4027c1ac-03a5-4918-b917-2c134741e4b5\") "
Jul 23 12:21:46.713819 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:21:46.713716    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4027c1ac-03a5-4918-b917-2c134741e4b5-tmp-dir\") pod \"4027c1ac-03a5-4918-b917-2c134741e4b5\" (UID: \"4027c1ac-03a5-4918-b917-2c134741e4b5\") "
Jul 23 12:21:46.713819 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:21:46.713737    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4027c1ac-03a5-4918-b917-2c134741e4b5-tls-certs\") pod \"4027c1ac-03a5-4918-b917-2c134741e4b5\" (UID: \"4027c1ac-03a5-4918-b917-2c134741e4b5\") "
Jul 23 12:21:46.713819 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:21:46.713758    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4027c1ac-03a5-4918-b917-2c134741e4b5-home\") pod \"4027c1ac-03a5-4918-b917-2c134741e4b5\" (UID: \"4027c1ac-03a5-4918-b917-2c134741e4b5\") "
Jul 23 12:21:46.714084 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:21:46.713931    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4027c1ac-03a5-4918-b917-2c134741e4b5-model-cache" (OuterVolumeSpecName: "model-cache") pod "4027c1ac-03a5-4918-b917-2c134741e4b5" (UID: "4027c1ac-03a5-4918-b917-2c134741e4b5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 12:21:46.714084 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:21:46.713955    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4027c1ac-03a5-4918-b917-2c134741e4b5-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4027c1ac-03a5-4918-b917-2c134741e4b5" (UID: "4027c1ac-03a5-4918-b917-2c134741e4b5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 12:21:46.714084 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:21:46.714063    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4027c1ac-03a5-4918-b917-2c134741e4b5-home" (OuterVolumeSpecName: "home") pod "4027c1ac-03a5-4918-b917-2c134741e4b5" (UID: "4027c1ac-03a5-4918-b917-2c134741e4b5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 12:21:46.715852 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:21:46.715828    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4027c1ac-03a5-4918-b917-2c134741e4b5-dshm" (OuterVolumeSpecName: "dshm") pod "4027c1ac-03a5-4918-b917-2c134741e4b5" (UID: "4027c1ac-03a5-4918-b917-2c134741e4b5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 12:21:46.715962 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:21:46.715936    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4027c1ac-03a5-4918-b917-2c134741e4b5-kube-api-access-d2xbc" (OuterVolumeSpecName: "kube-api-access-d2xbc") pod "4027c1ac-03a5-4918-b917-2c134741e4b5" (UID: "4027c1ac-03a5-4918-b917-2c134741e4b5"). InnerVolumeSpecName "kube-api-access-d2xbc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 12:21:46.716012 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:21:46.715979    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4027c1ac-03a5-4918-b917-2c134741e4b5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4027c1ac-03a5-4918-b917-2c134741e4b5" (UID: "4027c1ac-03a5-4918-b917-2c134741e4b5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 12:21:46.814969 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:21:46.814930    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4027c1ac-03a5-4918-b917-2c134741e4b5-tls-certs\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 23 12:21:46.814969 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:21:46.814960    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4027c1ac-03a5-4918-b917-2c134741e4b5-home\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 23 12:21:46.814969 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:21:46.814968    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4027c1ac-03a5-4918-b917-2c134741e4b5-model-cache\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 23 12:21:46.814969 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:21:46.814976    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4027c1ac-03a5-4918-b917-2c134741e4b5-dshm\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 23 12:21:46.815227 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:21:46.814985    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d2xbc\" (UniqueName: \"kubernetes.io/projected/4027c1ac-03a5-4918-b917-2c134741e4b5-kube-api-access-d2xbc\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 23 12:21:46.815227 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:21:46.815012    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4027c1ac-03a5-4918-b917-2c134741e4b5-tmp-dir\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 23 12:21:47.063247 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:21:47.063155    2563 generic.go:356] "Generic (PLEG): container finished" podID="4027c1ac-03a5-4918-b917-2c134741e4b5" containerID="bd7fce4ea6ad43eae7f752d5bdb2ce2ae0ff5238dab75d46bd8b618d0426285f" exitCode=0
Jul 23 12:21:47.063247 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:21:47.063214    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-290648/sub-test-model-290648-kserve-b6d574d89-crxpl"
Jul 23 12:21:47.063247 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:21:47.063226    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-290648/sub-test-model-290648-kserve-b6d574d89-crxpl" event={"ID":"4027c1ac-03a5-4918-b917-2c134741e4b5","Type":"ContainerDied","Data":"bd7fce4ea6ad43eae7f752d5bdb2ce2ae0ff5238dab75d46bd8b618d0426285f"}
Jul 23 12:21:47.063492 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:21:47.063263    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-290648/sub-test-model-290648-kserve-b6d574d89-crxpl" event={"ID":"4027c1ac-03a5-4918-b917-2c134741e4b5","Type":"ContainerDied","Data":"36b1cb0bb66282b66282fedfef6921908ff130886cb92ed27d2a513f01bd130c"}
Jul 23 12:21:47.063492 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:21:47.063282    2563 scope.go:117] "RemoveContainer" containerID="bd7fce4ea6ad43eae7f752d5bdb2ce2ae0ff5238dab75d46bd8b618d0426285f"
Jul 23 12:21:47.072594 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:21:47.072570    2563 scope.go:117] "RemoveContainer" containerID="bd7fce4ea6ad43eae7f752d5bdb2ce2ae0ff5238dab75d46bd8b618d0426285f"
Jul 23 12:21:47.073003 ip-10-0-141-229 kubenswrapper[2563]: E0723 12:21:47.072981    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd7fce4ea6ad43eae7f752d5bdb2ce2ae0ff5238dab75d46bd8b618d0426285f\": container with ID starting with bd7fce4ea6ad43eae7f752d5bdb2ce2ae0ff5238dab75d46bd8b618d0426285f not found: ID does not exist" containerID="bd7fce4ea6ad43eae7f752d5bdb2ce2ae0ff5238dab75d46bd8b618d0426285f"
Jul 23 12:21:47.073058 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:21:47.073013    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd7fce4ea6ad43eae7f752d5bdb2ce2ae0ff5238dab75d46bd8b618d0426285f"} err="failed to get container status \"bd7fce4ea6ad43eae7f752d5bdb2ce2ae0ff5238dab75d46bd8b618d0426285f\": rpc error: code = NotFound desc = could not find container \"bd7fce4ea6ad43eae7f752d5bdb2ce2ae0ff5238dab75d46bd8b618d0426285f\": container with ID starting with bd7fce4ea6ad43eae7f752d5bdb2ce2ae0ff5238dab75d46bd8b618d0426285f not found: ID does not exist"
Jul 23 12:21:47.084948 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:21:47.084911    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-290648/sub-test-model-290648-kserve-b6d574d89-crxpl"]
Jul 23 12:21:47.089471 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:21:47.089446    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-290648/sub-test-model-290648-kserve-b6d574d89-crxpl"]
Jul 23 12:21:47.470972 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:21:47.470933    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:21:47.570808 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:21:47.570770    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:21:48.077286 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:21:48.077255    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4027c1ac-03a5-4918-b917-2c134741e4b5" path="/var/lib/kubelet/pods/4027c1ac-03a5-4918-b917-2c134741e4b5/volumes"
Jul 23 12:21:50.552511 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:21:50.552466    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-290648/rate-test-model-290648-kserve-5675db7d67-sx7bd"
Jul 23 12:21:50.553053 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:21:50.552552    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-290648/rate-test-model-290648-kserve-5675db7d67-sx7bd"
Jul 23 12:21:50.565183 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:21:50.565153    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-290648/rate-test-model-290648-kserve-5675db7d67-sx7bd"
Jul 23 12:21:51.093649 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:21:51.093621    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-290648/rate-test-model-290648-kserve-5675db7d67-sx7bd"
Jul 23 12:21:56.517005 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:21:56.516961    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:21:56.619915 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:21:56.619881    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:22:13.875893 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:22:13.875818    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:22:18.407568 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:22:18.407535    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:22:45.775246 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:22:45.775211    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:22:45.838921 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:22:45.838883    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:22:58.420183 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:22:58.420152    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:23:01.349092 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:23:01.349060    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-290648/rate-test-model-290648-kserve-5675db7d67-sx7bd"]
Jul 23 12:23:01.373260 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:23:01.373228    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:23:02.377618 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:23:02.377586    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:23:03.402117 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:23:03.402083    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7832-139800632183344"]
Jul 23 12:23:03.402493 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:23:03.402470    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4027c1ac-03a5-4918-b917-2c134741e4b5" containerName="main"
Jul 23 12:23:03.402493 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:23:03.402482    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="4027c1ac-03a5-4918-b917-2c134741e4b5" containerName="main"
Jul 23 12:23:03.402574 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:23:03.402555    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="4027c1ac-03a5-4918-b917-2c134741e4b5" containerName="main"
Jul 23 12:23:03.409200 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:23:03.407992    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7832-139800632183344"
Jul 23 12:23:03.411277 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:23:03.411040    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-5d7qv\""
Jul 23 12:23:03.413361 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:23:03.413338    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7832-139800632183344"]
Jul 23 12:23:03.486660 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:23:03.486623    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4t2t\" (UniqueName: \"kubernetes.io/projected/76c92e2d-7195-425b-beec-efb80eb354f6-kube-api-access-k4t2t\") pod \"test-curl-7832-139800632183344\" (UID: \"76c92e2d-7195-425b-beec-efb80eb354f6\") " pod="opendatahub/test-curl-7832-139800632183344"
Jul 23 12:23:03.588076 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:23:03.588029    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4t2t\" (UniqueName: \"kubernetes.io/projected/76c92e2d-7195-425b-beec-efb80eb354f6-kube-api-access-k4t2t\") pod \"test-curl-7832-139800632183344\" (UID: \"76c92e2d-7195-425b-beec-efb80eb354f6\") " pod="opendatahub/test-curl-7832-139800632183344"
Jul 23 12:23:03.597592 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:23:03.597561    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4t2t\" (UniqueName: \"kubernetes.io/projected/76c92e2d-7195-425b-beec-efb80eb354f6-kube-api-access-k4t2t\") pod \"test-curl-7832-139800632183344\" (UID: \"76c92e2d-7195-425b-beec-efb80eb354f6\") " pod="opendatahub/test-curl-7832-139800632183344"
Jul 23 12:23:03.721067 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:23:03.720966    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7832-139800632183344"
Jul 23 12:23:03.855304 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:23:03.855262    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7832-139800632183344"]
Jul 23 12:23:03.859288 ip-10-0-141-229 kubenswrapper[2563]: W0723 12:23:03.859258    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod76c92e2d_7195_425b_beec_efb80eb354f6.slice/crio-ecfe93b02ffe17c24e7327a281693a374de34784439b197c63b3406052ffc745 WatchSource:0}: Error finding container ecfe93b02ffe17c24e7327a281693a374de34784439b197c63b3406052ffc745: Status 404 returned error can't find the container with id ecfe93b02ffe17c24e7327a281693a374de34784439b197c63b3406052ffc745
Jul 23 12:23:03.860580 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:23:03.860564    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 23 12:23:04.420518 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:23:04.420474    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7832-139800632183344" event={"ID":"76c92e2d-7195-425b-beec-efb80eb354f6","Type":"ContainerStarted","Data":"ecfe93b02ffe17c24e7327a281693a374de34784439b197c63b3406052ffc745"}
Jul 23 12:23:05.426519 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:23:05.426479    2563 generic.go:356] "Generic (PLEG): container finished" podID="76c92e2d-7195-425b-beec-efb80eb354f6" containerID="bffef47af172609351bd5bae311e9e0020cf94e3f5d7cb6dd4c906d504286eb9" exitCode=0
Jul 23 12:23:05.427037 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:23:05.426570    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7832-139800632183344" event={"ID":"76c92e2d-7195-425b-beec-efb80eb354f6","Type":"ContainerDied","Data":"bffef47af172609351bd5bae311e9e0020cf94e3f5d7cb6dd4c906d504286eb9"}
Jul 23 12:23:06.561799 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:23:06.561776    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7832-139800632183344"
Jul 23 12:23:06.582822 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:23:06.582790    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7832-139800632183344_76c92e2d-7195-425b-beec-efb80eb354f6/test-curl-7832-139800632183344/0.log"
Jul 23 12:23:06.606419 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:23:06.606387    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7832-139800632183344"]
Jul 23 12:23:06.608823 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:23:06.608801    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7832-139800632183344"]
Jul 23 12:23:06.716601 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:23:06.716506    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k4t2t\" (UniqueName: \"kubernetes.io/projected/76c92e2d-7195-425b-beec-efb80eb354f6-kube-api-access-k4t2t\") pod \"76c92e2d-7195-425b-beec-efb80eb354f6\" (UID: \"76c92e2d-7195-425b-beec-efb80eb354f6\") "
Jul 23 12:23:06.718810 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:23:06.718776    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76c92e2d-7195-425b-beec-efb80eb354f6-kube-api-access-k4t2t" (OuterVolumeSpecName: "kube-api-access-k4t2t") pod "76c92e2d-7195-425b-beec-efb80eb354f6" (UID: "76c92e2d-7195-425b-beec-efb80eb354f6"). InnerVolumeSpecName "kube-api-access-k4t2t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 12:23:06.817257 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:23:06.817223    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k4t2t\" (UniqueName: \"kubernetes.io/projected/76c92e2d-7195-425b-beec-efb80eb354f6-kube-api-access-k4t2t\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 23 12:23:07.437306 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:23:07.437277    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7832-139800632183344"
Jul 23 12:23:07.437488 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:23:07.437311    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ecfe93b02ffe17c24e7327a281693a374de34784439b197c63b3406052ffc745"
Jul 23 12:23:07.446892 ip-10-0-141-229 kubenswrapper[2563]: E0723 12:23:07.446865    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7832-139800632183344\" is forbidden: User \"system:node:ip-10-0-141-229.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-229.ec2.internal' and this object" podUID="76c92e2d-7195-425b-beec-efb80eb354f6" pod="opendatahub/test-curl-7832-139800632183344"
Jul 23 12:23:08.077549 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:23:08.077517    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="76c92e2d-7195-425b-beec-efb80eb354f6" path="/var/lib/kubelet/pods/76c92e2d-7195-425b-beec-efb80eb354f6/volumes"
Jul 23 12:23:15.697824 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:23:15.697789    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7832-139800632018960"]
Jul 23 12:23:15.698179 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:23:15.698148    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="76c92e2d-7195-425b-beec-efb80eb354f6" containerName="test-curl-7832-139800632183344"
Jul 23 12:23:15.698179 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:23:15.698160    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="76c92e2d-7195-425b-beec-efb80eb354f6" containerName="test-curl-7832-139800632183344"
Jul 23 12:23:15.698265 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:23:15.698229    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="76c92e2d-7195-425b-beec-efb80eb354f6" containerName="test-curl-7832-139800632183344"
Jul 23 12:23:15.701378 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:23:15.701359    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7832-139800632018960"
Jul 23 12:23:15.703848 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:23:15.703830    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-5d7qv\""
Jul 23 12:23:15.707267 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:23:15.707240    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7832-139800632018960"]
Jul 23 12:23:15.796485 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:23:15.796448    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7nlmv\" (UniqueName: \"kubernetes.io/projected/ac0ff15f-b1d9-4317-803d-f137d9c7193a-kube-api-access-7nlmv\") pod \"test-curl-7832-139800632018960\" (UID: \"ac0ff15f-b1d9-4317-803d-f137d9c7193a\") " pod="opendatahub/test-curl-7832-139800632018960"
Jul 23 12:23:15.897255 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:23:15.897223    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7nlmv\" (UniqueName: \"kubernetes.io/projected/ac0ff15f-b1d9-4317-803d-f137d9c7193a-kube-api-access-7nlmv\") pod \"test-curl-7832-139800632018960\" (UID: \"ac0ff15f-b1d9-4317-803d-f137d9c7193a\") " pod="opendatahub/test-curl-7832-139800632018960"
Jul 23 12:23:15.910513 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:23:15.910482    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7nlmv\" (UniqueName: \"kubernetes.io/projected/ac0ff15f-b1d9-4317-803d-f137d9c7193a-kube-api-access-7nlmv\") pod \"test-curl-7832-139800632018960\" (UID: \"ac0ff15f-b1d9-4317-803d-f137d9c7193a\") " pod="opendatahub/test-curl-7832-139800632018960"
Jul 23 12:23:16.011932 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:23:16.011855    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7832-139800632018960"
Jul 23 12:23:16.145353 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:23:16.145327    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7832-139800632018960"]
Jul 23 12:23:16.146736 ip-10-0-141-229 kubenswrapper[2563]: W0723 12:23:16.146706    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podac0ff15f_b1d9_4317_803d_f137d9c7193a.slice/crio-57376efc1711f95063104f41aebb434609fd4642709335266ef23dd895ca6150 WatchSource:0}: Error finding container 57376efc1711f95063104f41aebb434609fd4642709335266ef23dd895ca6150: Status 404 returned error can't find the container with id 57376efc1711f95063104f41aebb434609fd4642709335266ef23dd895ca6150
Jul 23 12:23:16.369205 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:23:16.369166    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-290648/rate-test-model-290648-kserve-5675db7d67-sx7bd" podUID="e565cc2d-74a0-47fe-be0c-d669f308e3ce" containerName="main" containerID="cri-o://a2db0d2129f782566d312baa7c5dcc26fd675e53e348f7260ff6e009673b21c8" gracePeriod=45
Jul 23 12:23:16.484336 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:23:16.484301    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7832-139800632018960" event={"ID":"ac0ff15f-b1d9-4317-803d-f137d9c7193a","Type":"ContainerStarted","Data":"57376efc1711f95063104f41aebb434609fd4642709335266ef23dd895ca6150"}
Jul 23 12:23:16.608096 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:23:16.608069    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-290648/rate-test-model-290648-kserve-5675db7d67-sx7bd"
Jul 23 12:23:16.704449 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:23:16.704358    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e565cc2d-74a0-47fe-be0c-d669f308e3ce-tmp-dir\") pod \"e565cc2d-74a0-47fe-be0c-d669f308e3ce\" (UID: \"e565cc2d-74a0-47fe-be0c-d669f308e3ce\") "
Jul 23 12:23:16.704449 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:23:16.704407    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wlp4w\" (UniqueName: \"kubernetes.io/projected/e565cc2d-74a0-47fe-be0c-d669f308e3ce-kube-api-access-wlp4w\") pod \"e565cc2d-74a0-47fe-be0c-d669f308e3ce\" (UID: \"e565cc2d-74a0-47fe-be0c-d669f308e3ce\") "
Jul 23 12:23:16.704449 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:23:16.704449    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e565cc2d-74a0-47fe-be0c-d669f308e3ce-dshm\") pod \"e565cc2d-74a0-47fe-be0c-d669f308e3ce\" (UID: \"e565cc2d-74a0-47fe-be0c-d669f308e3ce\") "
Jul 23 12:23:16.705023 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:23:16.704483    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e565cc2d-74a0-47fe-be0c-d669f308e3ce-model-cache\") pod \"e565cc2d-74a0-47fe-be0c-d669f308e3ce\" (UID: \"e565cc2d-74a0-47fe-be0c-d669f308e3ce\") "
Jul 23 12:23:16.705023 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:23:16.704541    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e565cc2d-74a0-47fe-be0c-d669f308e3ce-home\") pod \"e565cc2d-74a0-47fe-be0c-d669f308e3ce\" (UID: \"e565cc2d-74a0-47fe-be0c-d669f308e3ce\") "
Jul 23 12:23:16.705023 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:23:16.704609    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e565cc2d-74a0-47fe-be0c-d669f308e3ce-tls-certs\") pod \"e565cc2d-74a0-47fe-be0c-d669f308e3ce\" (UID: \"e565cc2d-74a0-47fe-be0c-d669f308e3ce\") "
Jul 23 12:23:16.705023 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:23:16.704629    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e565cc2d-74a0-47fe-be0c-d669f308e3ce-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e565cc2d-74a0-47fe-be0c-d669f308e3ce" (UID: "e565cc2d-74a0-47fe-be0c-d669f308e3ce"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 12:23:16.705023 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:23:16.704789    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e565cc2d-74a0-47fe-be0c-d669f308e3ce-model-cache" (OuterVolumeSpecName: "model-cache") pod "e565cc2d-74a0-47fe-be0c-d669f308e3ce" (UID: "e565cc2d-74a0-47fe-be0c-d669f308e3ce"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 12:23:16.705023 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:23:16.704857    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e565cc2d-74a0-47fe-be0c-d669f308e3ce-home" (OuterVolumeSpecName: "home") pod "e565cc2d-74a0-47fe-be0c-d669f308e3ce" (UID: "e565cc2d-74a0-47fe-be0c-d669f308e3ce"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 12:23:16.705023 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:23:16.704969    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e565cc2d-74a0-47fe-be0c-d669f308e3ce-tmp-dir\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 23 12:23:16.705023 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:23:16.704994    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e565cc2d-74a0-47fe-be0c-d669f308e3ce-model-cache\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 23 12:23:16.705023 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:23:16.705006    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e565cc2d-74a0-47fe-be0c-d669f308e3ce-home\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 23 12:23:16.706650 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:23:16.706627    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e565cc2d-74a0-47fe-be0c-d669f308e3ce-dshm" (OuterVolumeSpecName: "dshm") pod "e565cc2d-74a0-47fe-be0c-d669f308e3ce" (UID: "e565cc2d-74a0-47fe-be0c-d669f308e3ce"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 12:23:16.706810 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:23:16.706790    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e565cc2d-74a0-47fe-be0c-d669f308e3ce-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e565cc2d-74a0-47fe-be0c-d669f308e3ce" (UID: "e565cc2d-74a0-47fe-be0c-d669f308e3ce"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 12:23:16.706879 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:23:16.706793    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e565cc2d-74a0-47fe-be0c-d669f308e3ce-kube-api-access-wlp4w" (OuterVolumeSpecName: "kube-api-access-wlp4w") pod "e565cc2d-74a0-47fe-be0c-d669f308e3ce" (UID: "e565cc2d-74a0-47fe-be0c-d669f308e3ce"). InnerVolumeSpecName "kube-api-access-wlp4w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 12:23:16.806425 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:23:16.806393    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e565cc2d-74a0-47fe-be0c-d669f308e3ce-tls-certs\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 23 12:23:16.806425 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:23:16.806418    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wlp4w\" (UniqueName: \"kubernetes.io/projected/e565cc2d-74a0-47fe-be0c-d669f308e3ce-kube-api-access-wlp4w\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 23 12:23:16.806425 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:23:16.806428    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e565cc2d-74a0-47fe-be0c-d669f308e3ce-dshm\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 23 12:23:17.489696 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:23:17.489646    2563 generic.go:356] "Generic (PLEG): container finished" podID="e565cc2d-74a0-47fe-be0c-d669f308e3ce" containerID="a2db0d2129f782566d312baa7c5dcc26fd675e53e348f7260ff6e009673b21c8" exitCode=0
Jul 23 12:23:17.489892 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:23:17.489715    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-290648/rate-test-model-290648-kserve-5675db7d67-sx7bd"
Jul 23 12:23:17.489892 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:23:17.489732    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-290648/rate-test-model-290648-kserve-5675db7d67-sx7bd" event={"ID":"e565cc2d-74a0-47fe-be0c-d669f308e3ce","Type":"ContainerDied","Data":"a2db0d2129f782566d312baa7c5dcc26fd675e53e348f7260ff6e009673b21c8"}
Jul 23 12:23:17.489892 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:23:17.489771    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-290648/rate-test-model-290648-kserve-5675db7d67-sx7bd" event={"ID":"e565cc2d-74a0-47fe-be0c-d669f308e3ce","Type":"ContainerDied","Data":"043761b9dd8ab58370f25395c520c36eac5d1eebb6d28c59d8af2474fb431540"}
Jul 23 12:23:17.489892 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:23:17.489796    2563 scope.go:117] "RemoveContainer" containerID="a2db0d2129f782566d312baa7c5dcc26fd675e53e348f7260ff6e009673b21c8"
Jul 23 12:23:17.491488 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:23:17.491463    2563 generic.go:356] "Generic (PLEG): container finished" podID="ac0ff15f-b1d9-4317-803d-f137d9c7193a" containerID="d8ca3af5af5bd47609c4368d696dd4093bcc40e05645c6ff7ba39b0ad6bce738" exitCode=0
Jul 23 12:23:17.491591 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:23:17.491532    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7832-139800632018960" event={"ID":"ac0ff15f-b1d9-4317-803d-f137d9c7193a","Type":"ContainerDied","Data":"d8ca3af5af5bd47609c4368d696dd4093bcc40e05645c6ff7ba39b0ad6bce738"}
Jul 23 12:23:17.499650 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:23:17.499632    2563 scope.go:117] "RemoveContainer" containerID="a2db0d2129f782566d312baa7c5dcc26fd675e53e348f7260ff6e009673b21c8"
Jul 23 12:23:17.499940 ip-10-0-141-229 kubenswrapper[2563]: E0723 12:23:17.499917    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2db0d2129f782566d312baa7c5dcc26fd675e53e348f7260ff6e009673b21c8\": container with ID starting with a2db0d2129f782566d312baa7c5dcc26fd675e53e348f7260ff6e009673b21c8 not found: ID does not exist" containerID="a2db0d2129f782566d312baa7c5dcc26fd675e53e348f7260ff6e009673b21c8"
Jul 23 12:23:17.500000 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:23:17.499947    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2db0d2129f782566d312baa7c5dcc26fd675e53e348f7260ff6e009673b21c8"} err="failed to get container status \"a2db0d2129f782566d312baa7c5dcc26fd675e53e348f7260ff6e009673b21c8\": rpc error: code = NotFound desc = could not find container \"a2db0d2129f782566d312baa7c5dcc26fd675e53e348f7260ff6e009673b21c8\": container with ID starting with a2db0d2129f782566d312baa7c5dcc26fd675e53e348f7260ff6e009673b21c8 not found: ID does not exist"
Jul 23 12:23:17.523423 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:23:17.523386    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-290648/rate-test-model-290648-kserve-5675db7d67-sx7bd"]
Jul 23 12:23:17.525735 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:23:17.525713    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-290648/rate-test-model-290648-kserve-5675db7d67-sx7bd"]
Jul 23 12:23:18.079292 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:23:18.079246    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e565cc2d-74a0-47fe-be0c-d669f308e3ce" path="/var/lib/kubelet/pods/e565cc2d-74a0-47fe-be0c-d669f308e3ce/volumes"
Jul 23 12:23:18.629950 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:23:18.629922    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7832-139800632018960"
Jul 23 12:23:18.653978 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:23:18.653949    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7832-139800632018960_ac0ff15f-b1d9-4317-803d-f137d9c7193a/test-curl-7832-139800632018960/0.log"
Jul 23 12:23:18.677843 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:23:18.677812    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7832-139800632018960"]
Jul 23 12:23:18.682455 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:23:18.682429    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7832-139800632018960"]
Jul 23 12:23:18.722167 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:23:18.722137    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7nlmv\" (UniqueName: \"kubernetes.io/projected/ac0ff15f-b1d9-4317-803d-f137d9c7193a-kube-api-access-7nlmv\") pod \"ac0ff15f-b1d9-4317-803d-f137d9c7193a\" (UID: \"ac0ff15f-b1d9-4317-803d-f137d9c7193a\") "
Jul 23 12:23:18.724248 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:23:18.724220    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac0ff15f-b1d9-4317-803d-f137d9c7193a-kube-api-access-7nlmv" (OuterVolumeSpecName: "kube-api-access-7nlmv") pod "ac0ff15f-b1d9-4317-803d-f137d9c7193a" (UID: "ac0ff15f-b1d9-4317-803d-f137d9c7193a"). InnerVolumeSpecName "kube-api-access-7nlmv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 12:23:18.817753 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:23:18.817664    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7832-139800632018960"]
Jul 23 12:23:18.818054 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:23:18.818039    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e565cc2d-74a0-47fe-be0c-d669f308e3ce" containerName="main"
Jul 23 12:23:18.818099 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:23:18.818057    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="e565cc2d-74a0-47fe-be0c-d669f308e3ce" containerName="main"
Jul 23 12:23:18.818099 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:23:18.818068    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ac0ff15f-b1d9-4317-803d-f137d9c7193a" containerName="test-curl-7832-139800632018960"
Jul 23 12:23:18.818099 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:23:18.818076    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="ac0ff15f-b1d9-4317-803d-f137d9c7193a" containerName="test-curl-7832-139800632018960"
Jul 23 12:23:18.818187 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:23:18.818145    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="e565cc2d-74a0-47fe-be0c-d669f308e3ce" containerName="main"
Jul 23 12:23:18.818187 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:23:18.818154    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="ac0ff15f-b1d9-4317-803d-f137d9c7193a" containerName="test-curl-7832-139800632018960"
Jul 23 12:23:18.821191 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:23:18.821174    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7832-139800632018960"
Jul 23 12:23:18.823539 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:23:18.823519    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7nlmv\" (UniqueName: \"kubernetes.io/projected/ac0ff15f-b1d9-4317-803d-f137d9c7193a-kube-api-access-7nlmv\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 23 12:23:18.827950 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:23:18.827931    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7832-139800632018960"]
Jul 23 12:23:18.924449 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:23:18.924414    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xk2kf\" (UniqueName: \"kubernetes.io/projected/aedc3044-837e-4aee-b3c3-022c82eda585-kube-api-access-xk2kf\") pod \"test-curl-7832-139800632018960\" (UID: \"aedc3044-837e-4aee-b3c3-022c82eda585\") " pod="opendatahub/test-curl-7832-139800632018960"
Jul 23 12:23:19.025404 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:23:19.025349    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xk2kf\" (UniqueName: \"kubernetes.io/projected/aedc3044-837e-4aee-b3c3-022c82eda585-kube-api-access-xk2kf\") pod \"test-curl-7832-139800632018960\" (UID: \"aedc3044-837e-4aee-b3c3-022c82eda585\") " pod="opendatahub/test-curl-7832-139800632018960"
Jul 23 12:23:19.034707 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:23:19.034656    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xk2kf\" (UniqueName: \"kubernetes.io/projected/aedc3044-837e-4aee-b3c3-022c82eda585-kube-api-access-xk2kf\") pod \"test-curl-7832-139800632018960\" (UID: \"aedc3044-837e-4aee-b3c3-022c82eda585\") " pod="opendatahub/test-curl-7832-139800632018960"
Jul 23 12:23:19.131353 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:23:19.131321    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7832-139800632018960"
Jul 23 12:23:19.258946 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:23:19.258922    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7832-139800632018960"]
Jul 23 12:23:19.261507 ip-10-0-141-229 kubenswrapper[2563]: W0723 12:23:19.261475    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podaedc3044_837e_4aee_b3c3_022c82eda585.slice/crio-5c82863d9cb3b105f37521ba2a5d8e24085f65d8cf40a7c8c62cdee0d7afb403 WatchSource:0}: Error finding container 5c82863d9cb3b105f37521ba2a5d8e24085f65d8cf40a7c8c62cdee0d7afb403: Status 404 returned error can't find the container with id 5c82863d9cb3b105f37521ba2a5d8e24085f65d8cf40a7c8c62cdee0d7afb403
Jul 23 12:23:19.502111 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:23:19.502011    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7832-139800632018960" event={"ID":"aedc3044-837e-4aee-b3c3-022c82eda585","Type":"ContainerStarted","Data":"5c82863d9cb3b105f37521ba2a5d8e24085f65d8cf40a7c8c62cdee0d7afb403"}
Jul 23 12:23:19.503094 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:23:19.503074    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7832-139800632018960"
Jul 23 12:23:19.503094 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:23:19.503087    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="57376efc1711f95063104f41aebb434609fd4642709335266ef23dd895ca6150"
Jul 23 12:23:19.513663 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:23:19.513634    2563 status_manager.go:1027] "Pod was deleted and then recreated, skipping status update" pod="opendatahub/test-curl-7832-139800632018960" oldPodUID="ac0ff15f-b1d9-4317-803d-f137d9c7193a" podUID="aedc3044-837e-4aee-b3c3-022c82eda585"
Jul 23 12:23:20.077551 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:23:20.077510    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ac0ff15f-b1d9-4317-803d-f137d9c7193a" path="/var/lib/kubelet/pods/ac0ff15f-b1d9-4317-803d-f137d9c7193a/volumes"
Jul 23 12:23:20.509326 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:23:20.509294    2563 generic.go:356] "Generic (PLEG): container finished" podID="aedc3044-837e-4aee-b3c3-022c82eda585" containerID="2697ed0732a705d384ece32198941b8d2be8739e5270a12205e5f4b50b0ac134" exitCode=0
Jul 23 12:23:20.509745 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:23:20.509355    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7832-139800632018960" event={"ID":"aedc3044-837e-4aee-b3c3-022c82eda585","Type":"ContainerDied","Data":"2697ed0732a705d384ece32198941b8d2be8739e5270a12205e5f4b50b0ac134"}
Jul 23 12:23:21.653077 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:23:21.653046    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7832-139800632018960"
Jul 23 12:23:21.676400 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:23:21.676370    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7832-139800632018960_aedc3044-837e-4aee-b3c3-022c82eda585/test-curl-7832-139800632018960/0.log"
Jul 23 12:23:21.700047 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:23:21.700008    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7832-139800632018960"]
Jul 23 12:23:21.701695 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:23:21.701660    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7832-139800632018960"]
Jul 23 12:23:21.750234 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:23:21.750198    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xk2kf\" (UniqueName: \"kubernetes.io/projected/aedc3044-837e-4aee-b3c3-022c82eda585-kube-api-access-xk2kf\") pod \"aedc3044-837e-4aee-b3c3-022c82eda585\" (UID: \"aedc3044-837e-4aee-b3c3-022c82eda585\") "
Jul 23 12:23:21.752391 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:23:21.752365    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aedc3044-837e-4aee-b3c3-022c82eda585-kube-api-access-xk2kf" (OuterVolumeSpecName: "kube-api-access-xk2kf") pod "aedc3044-837e-4aee-b3c3-022c82eda585" (UID: "aedc3044-837e-4aee-b3c3-022c82eda585"). InnerVolumeSpecName "kube-api-access-xk2kf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 12:23:21.851578 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:23:21.851503    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xk2kf\" (UniqueName: \"kubernetes.io/projected/aedc3044-837e-4aee-b3c3-022c82eda585-kube-api-access-xk2kf\") on node \"ip-10-0-141-229.ec2.internal\" DevicePath \"\""
Jul 23 12:23:22.078124 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:23:22.078086    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aedc3044-837e-4aee-b3c3-022c82eda585" path="/var/lib/kubelet/pods/aedc3044-837e-4aee-b3c3-022c82eda585/volumes"
Jul 23 12:23:22.485512 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:23:22.485474    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:23:22.520873 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:23:22.520849    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7832-139800632018960"
Jul 23 12:23:22.521030 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:23:22.520904    2563 scope.go:117] "RemoveContainer" containerID="2697ed0732a705d384ece32198941b8d2be8739e5270a12205e5f4b50b0ac134"
Jul 23 12:23:22.572171 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:23:22.572138    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:23:32.530737 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:23:32.530699    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:23:36.985940 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:23:36.985858    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:23:37.412969 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:23:37.412927    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:23:43.473058 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:23:43.473013    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:23:43.555018 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:23:43.554962    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:23:43.772793 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:23:43.772709    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:23:44.025275 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:23:44.025180    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:23:44.792630 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:23:44.792595    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:23:44.890312 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:23:44.890279    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:23:54.679910 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:23:54.679880    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:23:59.244234 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:23:59.244186    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:23:59.640354 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:23:59.640313    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:24:06.076586 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:24:06.076554    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:24:06.173974 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:24:06.173939    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:24:06.350795 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:24:06.350701    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:24:06.585107 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:24:06.585061    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:24:13.036199 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:24:13.036160    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-78746d79b5-rrm4t_4f71af9a-2a14-4316-bb1f-194520cb518b/manager/0.log"
Jul 23 12:24:13.535778 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:24:13.535750    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7c7646c6cc-qj2nv_feaa1c4b-4dad-47ea-9511-95f0d907dc27/copy-manifests/0.log"
Jul 23 12:24:13.542272 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:24:13.542239    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7c7646c6cc-qj2nv_feaa1c4b-4dad-47ea-9511-95f0d907dc27/manager/0.log"
Jul 23 12:24:14.076939 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:24:14.076912    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-64c4fb9c4-25vlh_01034321-2fa1-4709-bbf0-23d21b05fb49/maas-api/0.log"
Jul 23 12:24:16.168668 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:24:16.168636    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-gxbsp_be20ca05-298e-47c1-9e10-e36c4200b713/kuadrant-console-plugin/0.log"
Jul 23 12:24:17.308481 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:24:17.308449    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5b7f786dc8-5gcr8_f7bb354b-f7b9-4a34-a239-4e985a64ce1c/kube-auth-proxy/0.log"
Jul 23 12:24:17.688126 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:24:17.688098    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7677b774bb-8mlt4_bfa0456d-4cdc-431d-b86b-075f929d20de/payload-pre-processing/0.log"
Jul 23 12:24:17.814938 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:24:17.814906    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5bf47cd988-qz6rk_0cca871f-490f-49cc-8782-a47403080a0d/payload-processing/0.log"
Jul 23 12:24:18.310828 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:24:18.310800    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-8827l_a431e4db-9a2b-47fe-94ff-4b92aed0b270/main/0.log"
Jul 23 12:24:18.434024 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:24:18.433992    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-gfcpc_20e400ab-125c-4414-b5c1-deb1455a5871/main/0.log"
Jul 23 12:24:18.804837 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:24:18.804803    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-f2pzs_906a2b44-6eee-47d1-9f8c-6657926fd9be/main/0.log"
Jul 23 12:24:18.925866 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:24:18.925818    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-x9l6l_e0e9242c-b071-4bf2-aa3e-1f176d0f5ae5/main/0.log"
Jul 23 12:24:25.329606 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:24:25.329568    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:24:25.424105 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:24:25.424059    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:24:28.384440 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:24:28.384409    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-cvbbt_7a0946cf-0f04-49d5-ab74-6f64a48f8e5d/global-pull-secret-syncer/0.log"
Jul 23 12:24:28.438489 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:24:28.438455    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log"
Jul 23 12:24:28.593380 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:24:28.593343    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-141-229.ec2.internal_192fe6abc0214ce0c06c38f944481f45/haproxy/0.log"
Jul 23 12:24:34.152025 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:24:34.151994    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-gxbsp_be20ca05-298e-47c1-9e10-e36c4200b713/kuadrant-console-plugin/0.log"
Jul 23 12:24:36.852099 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:24:36.852063    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-8564b6f9f6-9wwdl_a3b3aca3-c21e-4f24-be22-4c626e03bfbf/metrics-server/0.log"
Jul 23 12:24:36.893098 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:24:36.893069    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-7b7b7bc8bf-4tn6c_af96bf6b-bf0e-4dc4-8a34-71f0955c4bce/monitoring-plugin/0.log"
Jul 23 12:24:36.929252 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:24:36.929222    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-kmbvx_d79b19eb-59df-4f8c-affe-c67eb5e142aa/node-exporter/0.log"
Jul 23 12:24:36.956247 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:24:36.956216    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-kmbvx_d79b19eb-59df-4f8c-affe-c67eb5e142aa/kube-rbac-proxy/0.log"
Jul 23 12:24:36.981967 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:24:36.981947    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-kmbvx_d79b19eb-59df-4f8c-affe-c67eb5e142aa/init-textfile/0.log"
Jul 23 12:24:37.269838 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:24:37.269806    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_1de24f23-477c-4fbb-964a-7802ee86612e/prometheus/0.log"
Jul 23 12:24:37.291746 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:24:37.291721    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_1de24f23-477c-4fbb-964a-7802ee86612e/config-reloader/0.log"
Jul 23 12:24:37.314037 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:24:37.314015    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_1de24f23-477c-4fbb-964a-7802ee86612e/thanos-sidecar/0.log"
Jul 23 12:24:37.335503 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:24:37.335482    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_1de24f23-477c-4fbb-964a-7802ee86612e/kube-rbac-proxy-web/0.log"
Jul 23 12:24:37.358925 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:24:37.358889    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_1de24f23-477c-4fbb-964a-7802ee86612e/kube-rbac-proxy/0.log"
Jul 23 12:24:37.381199 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:24:37.381173    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_1de24f23-477c-4fbb-964a-7802ee86612e/kube-rbac-proxy-thanos/0.log"
Jul 23 12:24:37.407956 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:24:37.407933    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_1de24f23-477c-4fbb-964a-7802ee86612e/init-config-reloader/0.log"
Jul 23 12:24:37.636233 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:24:37.636208    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7dfbd954f7-jjdbk_0b575b60-75a2-4539-bc75-08e2c621bc0b/thanos-query/0.log"
Jul 23 12:24:37.667308 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:24:37.667282    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7dfbd954f7-jjdbk_0b575b60-75a2-4539-bc75-08e2c621bc0b/kube-rbac-proxy-web/0.log"
Jul 23 12:24:37.688338 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:24:37.688313    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7dfbd954f7-jjdbk_0b575b60-75a2-4539-bc75-08e2c621bc0b/kube-rbac-proxy/0.log"
Jul 23 12:24:37.709505 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:24:37.709485    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7dfbd954f7-jjdbk_0b575b60-75a2-4539-bc75-08e2c621bc0b/prom-label-proxy/0.log"
Jul 23 12:24:37.730418 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:24:37.730395    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7dfbd954f7-jjdbk_0b575b60-75a2-4539-bc75-08e2c621bc0b/kube-rbac-proxy-rules/0.log"
Jul 23 12:24:37.753151 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:24:37.753122    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7dfbd954f7-jjdbk_0b575b60-75a2-4539-bc75-08e2c621bc0b/kube-rbac-proxy-metrics/0.log"
Jul 23 12:24:38.752820 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:24:38.752781    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-5jgf7/perf-node-gather-daemonset-4jqls"]
Jul 23 12:24:38.753236 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:24:38.753167    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aedc3044-837e-4aee-b3c3-022c82eda585" containerName="test-curl-7832-139800632018960"
Jul 23 12:24:38.753236 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:24:38.753180    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="aedc3044-837e-4aee-b3c3-022c82eda585" containerName="test-curl-7832-139800632018960"
Jul 23 12:24:38.753326 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:24:38.753249    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="aedc3044-837e-4aee-b3c3-022c82eda585" containerName="test-curl-7832-139800632018960"
Jul 23 12:24:38.756353 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:24:38.756332    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5jgf7/perf-node-gather-daemonset-4jqls"
Jul 23 12:24:38.758823 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:24:38.758801    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-5jgf7\"/\"kube-root-ca.crt\""
Jul 23 12:24:38.758823 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:24:38.758813    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-5jgf7\"/\"default-dockercfg-st8ws\""
Jul 23 12:24:38.759007 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:24:38.758822    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-5jgf7\"/\"openshift-service-ca.crt\""
Jul 23 12:24:38.763736 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:24:38.763715    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-5jgf7/perf-node-gather-daemonset-4jqls"]
Jul 23 12:24:38.823885 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:24:38.823841    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/7439e864-9663-4c4b-8480-dc3163a585ce-proc\") pod \"perf-node-gather-daemonset-4jqls\" (UID: \"7439e864-9663-4c4b-8480-dc3163a585ce\") " pod="openshift-must-gather-5jgf7/perf-node-gather-daemonset-4jqls"
Jul 23 12:24:38.823885 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:24:38.823882    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/7439e864-9663-4c4b-8480-dc3163a585ce-podres\") pod \"perf-node-gather-daemonset-4jqls\" (UID: \"7439e864-9663-4c4b-8480-dc3163a585ce\") " pod="openshift-must-gather-5jgf7/perf-node-gather-daemonset-4jqls"
Jul 23 12:24:38.824104 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:24:38.823905    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8krw6\" (UniqueName: \"kubernetes.io/projected/7439e864-9663-4c4b-8480-dc3163a585ce-kube-api-access-8krw6\") pod \"perf-node-gather-daemonset-4jqls\" (UID: \"7439e864-9663-4c4b-8480-dc3163a585ce\") " pod="openshift-must-gather-5jgf7/perf-node-gather-daemonset-4jqls"
Jul 23 12:24:38.824104 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:24:38.824038    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7439e864-9663-4c4b-8480-dc3163a585ce-sys\") pod \"perf-node-gather-daemonset-4jqls\" (UID: \"7439e864-9663-4c4b-8480-dc3163a585ce\") " pod="openshift-must-gather-5jgf7/perf-node-gather-daemonset-4jqls"
Jul 23 12:24:38.824170 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:24:38.824104    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7439e864-9663-4c4b-8480-dc3163a585ce-lib-modules\") pod \"perf-node-gather-daemonset-4jqls\" (UID: \"7439e864-9663-4c4b-8480-dc3163a585ce\") " pod="openshift-must-gather-5jgf7/perf-node-gather-daemonset-4jqls"
Jul 23 12:24:38.925517 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:24:38.925477    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7439e864-9663-4c4b-8480-dc3163a585ce-lib-modules\") pod \"perf-node-gather-daemonset-4jqls\" (UID: \"7439e864-9663-4c4b-8480-dc3163a585ce\") " pod="openshift-must-gather-5jgf7/perf-node-gather-daemonset-4jqls"
Jul 23 12:24:38.925753 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:24:38.925527    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/7439e864-9663-4c4b-8480-dc3163a585ce-proc\") pod \"perf-node-gather-daemonset-4jqls\" (UID: \"7439e864-9663-4c4b-8480-dc3163a585ce\") " pod="openshift-must-gather-5jgf7/perf-node-gather-daemonset-4jqls"
Jul 23 12:24:38.925753 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:24:38.925544    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/7439e864-9663-4c4b-8480-dc3163a585ce-podres\") pod \"perf-node-gather-daemonset-4jqls\" (UID: \"7439e864-9663-4c4b-8480-dc3163a585ce\") " pod="openshift-must-gather-5jgf7/perf-node-gather-daemonset-4jqls"
Jul 23 12:24:38.925753 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:24:38.925558    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8krw6\" (UniqueName: \"kubernetes.io/projected/7439e864-9663-4c4b-8480-dc3163a585ce-kube-api-access-8krw6\") pod \"perf-node-gather-daemonset-4jqls\" (UID: \"7439e864-9663-4c4b-8480-dc3163a585ce\") " pod="openshift-must-gather-5jgf7/perf-node-gather-daemonset-4jqls"
Jul 23 12:24:38.925753 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:24:38.925618    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/7439e864-9663-4c4b-8480-dc3163a585ce-proc\") pod \"perf-node-gather-daemonset-4jqls\" (UID: \"7439e864-9663-4c4b-8480-dc3163a585ce\") " pod="openshift-must-gather-5jgf7/perf-node-gather-daemonset-4jqls"
Jul 23 12:24:38.925753 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:24:38.925621    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7439e864-9663-4c4b-8480-dc3163a585ce-sys\") pod \"perf-node-gather-daemonset-4jqls\" (UID: \"7439e864-9663-4c4b-8480-dc3163a585ce\") " pod="openshift-must-gather-5jgf7/perf-node-gather-daemonset-4jqls"
Jul 23 12:24:38.925753 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:24:38.925658    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7439e864-9663-4c4b-8480-dc3163a585ce-lib-modules\") pod \"perf-node-gather-daemonset-4jqls\" (UID: \"7439e864-9663-4c4b-8480-dc3163a585ce\") " pod="openshift-must-gather-5jgf7/perf-node-gather-daemonset-4jqls"
Jul 23 12:24:38.925753 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:24:38.925670    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/7439e864-9663-4c4b-8480-dc3163a585ce-podres\") pod \"perf-node-gather-daemonset-4jqls\" (UID: \"7439e864-9663-4c4b-8480-dc3163a585ce\") " pod="openshift-must-gather-5jgf7/perf-node-gather-daemonset-4jqls"
Jul 23 12:24:38.925753 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:24:38.925663    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7439e864-9663-4c4b-8480-dc3163a585ce-sys\") pod \"perf-node-gather-daemonset-4jqls\" (UID: \"7439e864-9663-4c4b-8480-dc3163a585ce\") " pod="openshift-must-gather-5jgf7/perf-node-gather-daemonset-4jqls"
Jul 23 12:24:38.934292 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:24:38.934250    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8krw6\" (UniqueName: \"kubernetes.io/projected/7439e864-9663-4c4b-8480-dc3163a585ce-kube-api-access-8krw6\") pod \"perf-node-gather-daemonset-4jqls\" (UID: \"7439e864-9663-4c4b-8480-dc3163a585ce\") " pod="openshift-must-gather-5jgf7/perf-node-gather-daemonset-4jqls"
Jul 23 12:24:38.957790 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:24:38.957762    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-7bc7489cb5-b7zjr_3320cd4a-2c2f-49fe-99fd-acd7be999bf1/networking-console-plugin/0.log"
Jul 23 12:24:39.066831 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:24:39.066731    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5jgf7/perf-node-gather-daemonset-4jqls"
Jul 23 12:24:39.198342 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:24:39.198318    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-5jgf7/perf-node-gather-daemonset-4jqls"]
Jul 23 12:24:39.200463 ip-10-0-141-229 kubenswrapper[2563]: W0723 12:24:39.200432    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod7439e864_9663_4c4b_8480_dc3163a585ce.slice/crio-05c582a8bf3d29371c360907b1f11b41ff9659d5bd44e84b6812ed47dc38314d WatchSource:0}: Error finding container 05c582a8bf3d29371c360907b1f11b41ff9659d5bd44e84b6812ed47dc38314d: Status 404 returned error can't find the container with id 05c582a8bf3d29371c360907b1f11b41ff9659d5bd44e84b6812ed47dc38314d
Jul 23 12:24:39.878044 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:24:39.878008    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5jgf7/perf-node-gather-daemonset-4jqls" event={"ID":"7439e864-9663-4c4b-8480-dc3163a585ce","Type":"ContainerStarted","Data":"68490c3741d30e1c40df178cdf67f2c26a242ae84d0cc77bc97e17a25d651e29"}
Jul 23 12:24:39.878543 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:24:39.878067    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5jgf7/perf-node-gather-daemonset-4jqls" event={"ID":"7439e864-9663-4c4b-8480-dc3163a585ce","Type":"ContainerStarted","Data":"05c582a8bf3d29371c360907b1f11b41ff9659d5bd44e84b6812ed47dc38314d"}
Jul 23 12:24:39.878543 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:24:39.878122    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-5jgf7/perf-node-gather-daemonset-4jqls"
Jul 23 12:24:39.898215 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:24:39.898162    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-5jgf7/perf-node-gather-daemonset-4jqls" podStartSLOduration=1.8981464909999999 podStartE2EDuration="1.898146491s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 12:24:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 12:24:39.896091998 +0000 UTC m=+2934.289029785" watchObservedRunningTime="2026-07-23 12:24:39.898146491 +0000 UTC m=+2934.291084277"
Jul 23 12:24:40.125589 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:24:40.125562    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-5bc67dcd46-nfpg5_2ab262b2-8386-4891-86d0-1620a18357c6/download-server/0.log"
Jul 23 12:24:41.890721 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:24:41.890678    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-t2l85_804c9466-58e2-4986-b0bf-2e85cba3d0e4/dns/0.log"
Jul 23 12:24:41.911495 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:24:41.911469    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-t2l85_804c9466-58e2-4986-b0bf-2e85cba3d0e4/kube-rbac-proxy/0.log"
Jul 23 12:24:41.933553 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:24:41.933516    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-dmnft_5ef3b70f-a117-4dcd-9380-5c5d36e596c0/dns-node-resolver/0.log"
Jul 23 12:24:42.124394 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:24:42.124346    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rs9k_73f77fae-fdff-4232-a26a-e5749f7df186/konnectivity-agent/0.log" limit=1024
Jul 23 12:24:42.545429 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:24:42.545395    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-c78dd7fdc-z2h9v_9faf18f6-bd93-4f36-999b-ddb434ba06be/registry/0.log"
Jul 23 12:24:42.618109 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:24:42.618076    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-dg4zg_2ba42487-6ccf-4e15-93df-65f0623c41ba/node-ca/0.log"
Jul 23 12:24:43.872788 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:24:43.872752    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5b7f786dc8-5gcr8_f7bb354b-f7b9-4a34-a239-4e985a64ce1c/kube-auth-proxy/0.log"
Jul 23 12:24:44.021139 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:24:44.021102    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7677b774bb-8mlt4_bfa0456d-4cdc-431d-b86b-075f929d20de/payload-pre-processing/0.log"
Jul 23 12:24:44.051442 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:24:44.051410    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5bf47cd988-qz6rk_0cca871f-490f-49cc-8782-a47403080a0d/payload-processing/0.log"
Jul 23 12:24:44.788138 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:24:44.788100    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-d5p84_595ac104-73d9-408b-b1e4-923312a4f5e7/serve-healthcheck-canary/0.log"
Jul 23 12:24:45.457502 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:24:45.457473    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-r4f2b_867645c0-1161-4522-b68f-f597f275ed37/kube-rbac-proxy/0.log"
Jul 23 12:24:45.482074 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:24:45.482043    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-r4f2b_867645c0-1161-4522-b68f-f597f275ed37/exporter/0.log"
Jul 23 12:24:45.502091 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:24:45.502067    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-r4f2b_867645c0-1161-4522-b68f-f597f275ed37/extractor/0.log"
Jul 23 12:24:45.892758 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:24:45.892731    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-5jgf7/perf-node-gather-daemonset-4jqls"
Jul 23 12:24:48.185432 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:24:48.185399    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-78746d79b5-rrm4t_4f71af9a-2a14-4316-bb1f-194520cb518b/manager/0.log"
Jul 23 12:24:48.405401 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:24:48.405372    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7c7646c6cc-qj2nv_feaa1c4b-4dad-47ea-9511-95f0d907dc27/manager/0.log"
Jul 23 12:24:48.431994 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:24:48.431968    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7c7646c6cc-qj2nv_feaa1c4b-4dad-47ea-9511-95f0d907dc27/copy-manifests/0.log"
Jul 23 12:24:49.885067 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:24:49.885031    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-544c65c659-4msr2_a558bac7-2851-4f03-ae33-76c40027ad4e/manager/0.log"
Jul 23 12:24:50.488231 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:24:50.488197    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5b7f786dc8-5gcr8_f7bb354b-f7b9-4a34-a239-4e985a64ce1c/kube-auth-proxy/0.log"
Jul 23 12:24:50.582056 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:24:50.582028    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7677b774bb-8mlt4_bfa0456d-4cdc-431d-b86b-075f929d20de/payload-pre-processing/0.log"
Jul 23 12:24:50.602009 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:24:50.601981    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5bf47cd988-qz6rk_0cca871f-490f-49cc-8782-a47403080a0d/payload-processing/0.log"
Jul 23 12:24:52.042729 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:24:52.042702    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5b7f786dc8-5gcr8_f7bb354b-f7b9-4a34-a239-4e985a64ce1c/kube-auth-proxy/0.log"
Jul 23 12:24:52.149621 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:24:52.149590    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7677b774bb-8mlt4_bfa0456d-4cdc-431d-b86b-075f929d20de/payload-pre-processing/0.log"
Jul 23 12:24:52.173149 ip-10-0-141-229 kubenswrapper[2563]: I0723 12:24:52.173116    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5bf47cd988-qz6rk_0cca871f-490f-49cc-8782-a47403080a0d/payload-processing/0.log"