Jul 23 17:05:04.119513 ip-10-0-129-216 systemd[1]: Starting Kubernetes Kubelet...
Jul 23 17:05:04.492789 ip-10-0-129-216 kubenswrapper[2577]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 23 17:05:04.492789 ip-10-0-129-216 kubenswrapper[2577]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 23 17:05:04.492789 ip-10-0-129-216 kubenswrapper[2577]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 23 17:05:04.492789 ip-10-0-129-216 kubenswrapper[2577]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 23 17:05:04.492789 ip-10-0-129-216 kubenswrapper[2577]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 23 17:05:04.493456 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.492853    2577 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 23 17:05:04.496393 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.496378    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 23 17:05:04.496393 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.496392    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 23 17:05:04.496453 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.496396    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 23 17:05:04.496453 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.496399    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 23 17:05:04.496453 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.496401    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 23 17:05:04.496453 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.496404    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 23 17:05:04.496453 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.496407    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 23 17:05:04.496453 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.496410    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 23 17:05:04.496453 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.496413    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 23 17:05:04.496453 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.496416    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 23 17:05:04.496453 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.496419    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 23 17:05:04.496453 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.496421    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 23 17:05:04.496453 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.496424    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 23 17:05:04.496453 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.496427    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 23 17:05:04.496453 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.496429    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 23 17:05:04.496453 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.496432    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 23 17:05:04.496453 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.496435    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 23 17:05:04.496453 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.496437    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 23 17:05:04.496453 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.496440    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 23 17:05:04.496453 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.496442    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 23 17:05:04.496453 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.496445    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 23 17:05:04.496453 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.496447    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 23 17:05:04.496939 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.496450    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 23 17:05:04.496939 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.496452    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 23 17:05:04.496939 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.496455    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 23 17:05:04.496939 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.496458    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 23 17:05:04.496939 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.496462    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 23 17:05:04.496939 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.496465    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 23 17:05:04.496939 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.496467    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 23 17:05:04.496939 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.496470    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 23 17:05:04.496939 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.496472    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 23 17:05:04.496939 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.496475    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 23 17:05:04.496939 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.496478    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 23 17:05:04.496939 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.496482    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 23 17:05:04.496939 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.496486    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 23 17:05:04.496939 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.496489    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 23 17:05:04.496939 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.496492    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 23 17:05:04.496939 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.496494    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 23 17:05:04.496939 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.496497    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 23 17:05:04.496939 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.496499    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 23 17:05:04.496939 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.496502    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 23 17:05:04.497428 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.496504    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 23 17:05:04.497428 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.496507    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 23 17:05:04.497428 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.496509    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 23 17:05:04.497428 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.496513    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 23 17:05:04.497428 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.496515    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 23 17:05:04.497428 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.496518    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 23 17:05:04.497428 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.496520    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 23 17:05:04.497428 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.496523    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 23 17:05:04.497428 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.496525    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 23 17:05:04.497428 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.496529    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 23 17:05:04.497428 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.496533    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 23 17:05:04.497428 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.496536    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 23 17:05:04.497428 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.496538    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 23 17:05:04.497428 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.496542    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 23 17:05:04.497428 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.496546    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 23 17:05:04.497428 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.496548    2577 feature_gate.go:328] unrecognized feature gate: Example
Jul 23 17:05:04.497428 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.496551    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 23 17:05:04.497428 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.496553    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 23 17:05:04.497428 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.496556    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 23 17:05:04.497875 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.496559    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 23 17:05:04.497875 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.496561    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 23 17:05:04.497875 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.496564    2577 feature_gate.go:328] unrecognized feature gate: Example2
Jul 23 17:05:04.497875 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.496566    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 23 17:05:04.497875 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.496569    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 23 17:05:04.497875 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.496572    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 23 17:05:04.497875 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.496574    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 23 17:05:04.497875 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.496576    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 23 17:05:04.497875 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.496579    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 23 17:05:04.497875 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.496581    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 23 17:05:04.497875 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.496584    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 23 17:05:04.497875 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.496586    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 23 17:05:04.497875 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.496589    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 23 17:05:04.497875 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.496594    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 23 17:05:04.497875 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.496597    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 23 17:05:04.497875 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.496600    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 23 17:05:04.497875 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.496603    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 23 17:05:04.497875 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.496606    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 23 17:05:04.497875 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.496609    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 23 17:05:04.498333 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.496612    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 23 17:05:04.498333 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.496615    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 23 17:05:04.498333 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.496618    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 23 17:05:04.498333 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.496620    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 23 17:05:04.498333 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.496623    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 23 17:05:04.498333 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.496626    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 23 17:05:04.498333 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.496628    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 23 17:05:04.498333 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.496631    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 23 17:05:04.498333 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.496634    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 23 17:05:04.498333 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.496637    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 23 17:05:04.498333 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.496640    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 23 17:05:04.498333 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.496642    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 23 17:05:04.498333 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.496645    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 23 17:05:04.498333 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.496647    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 23 17:05:04.498333 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.496650    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 23 17:05:04.498333 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.496653    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 23 17:05:04.498333 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.496655    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 23 17:05:04.498333 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.496658    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 23 17:05:04.498333 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.496660    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 23 17:05:04.498333 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.496662    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 23 17:05:04.498846 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.497351    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 23 17:05:04.498846 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.497359    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 23 17:05:04.498846 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.497362    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 23 17:05:04.498846 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.497364    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 23 17:05:04.498846 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.497367    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 23 17:05:04.498846 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.497370    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 23 17:05:04.498846 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.497373    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 23 17:05:04.498846 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.497375    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 23 17:05:04.498846 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.497378    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 23 17:05:04.498846 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.497382    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 23 17:05:04.498846 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.497385    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 23 17:05:04.498846 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.497388    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 23 17:05:04.498846 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.497391    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 23 17:05:04.498846 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.497393    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 23 17:05:04.498846 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.497395    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 23 17:05:04.498846 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.497398    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 23 17:05:04.498846 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.497401    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 23 17:05:04.498846 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.497403    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 23 17:05:04.498846 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.497405    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 23 17:05:04.498846 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.497408    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 23 17:05:04.499374 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.497411    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 23 17:05:04.499374 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.497414    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 23 17:05:04.499374 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.497416    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 23 17:05:04.499374 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.497419    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 23 17:05:04.499374 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.497422    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 23 17:05:04.499374 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.497424    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 23 17:05:04.499374 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.497427    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 23 17:05:04.499374 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.497431    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 23 17:05:04.499374 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.497433    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 23 17:05:04.499374 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.497436    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 23 17:05:04.499374 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.497439    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 23 17:05:04.499374 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.497441    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 23 17:05:04.499374 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.497444    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 23 17:05:04.499374 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.497446    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 23 17:05:04.499374 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.497448    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 23 17:05:04.499374 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.497451    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 23 17:05:04.499374 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.497454    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 23 17:05:04.499374 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.497456    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 23 17:05:04.499374 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.497459    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 23 17:05:04.499828 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.497462    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 23 17:05:04.499828 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.497464    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 23 17:05:04.499828 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.497467    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 23 17:05:04.499828 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.497470    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 23 17:05:04.499828 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.497473    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 23 17:05:04.499828 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.497476    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 23 17:05:04.499828 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.497478    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 23 17:05:04.499828 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.497481    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 23 17:05:04.499828 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.497483    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 23 17:05:04.499828 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.497487    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 23 17:05:04.499828 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.497489    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 23 17:05:04.499828 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.497492    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 23 17:05:04.499828 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.497494    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 23 17:05:04.499828 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.497497    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 23 17:05:04.499828 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.497500    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 23 17:05:04.499828 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.497502    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 23 17:05:04.499828 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.497505    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 23 17:05:04.499828 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.497507    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 23 17:05:04.499828 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.497510    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 23 17:05:04.499828 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.497518    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 23 17:05:04.500336 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.497521    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 23 17:05:04.500336 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.497523    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 23 17:05:04.500336 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.497526    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 23 17:05:04.500336 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.497530    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 23 17:05:04.500336 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.497534    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 23 17:05:04.500336 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.497537    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 23 17:05:04.500336 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.497540    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 23 17:05:04.500336 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.497543    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 23 17:05:04.500336 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.497545    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 23 17:05:04.500336 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.497548    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 23 17:05:04.500336 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.497551    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 23 17:05:04.500336 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.497554    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 23 17:05:04.500336 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.497556    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 23 17:05:04.500336 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.497559    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 23 17:05:04.500336 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.497561    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 23 17:05:04.500336 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.497564    2577 feature_gate.go:328] unrecognized feature gate: Example
Jul 23 17:05:04.500336 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.497567    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 23 17:05:04.500336 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.497569    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 23 17:05:04.500336 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.497572    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 23 17:05:04.500336 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.497574    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 23 17:05:04.500819 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.497577    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 23 17:05:04.500819 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.497579    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 23 17:05:04.500819 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.497583    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 23 17:05:04.500819 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.497586    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 23 17:05:04.500819 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.497589    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 23 17:05:04.500819 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.497592    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 23 17:05:04.500819 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.497594    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 23 17:05:04.500819 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.497597    2577 feature_gate.go:328] unrecognized feature gate: Example2
Jul 23 17:05:04.500819 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.497599    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 23 17:05:04.500819 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.497602    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 23 17:05:04.500819 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.497604    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 23 17:05:04.500819 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.497608    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 23 17:05:04.500819 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.497610    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 23 17:05:04.500819 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.497613    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 23 17:05:04.500819 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.497615    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 23 17:05:04.500819 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.497618    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 23 17:05:04.500819 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.497620    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 23 17:05:04.500819 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.497624    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 23 17:05:04.500819 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.497627    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 23 17:05:04.501289 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.497630    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 23 17:05:04.501289 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.497988    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 23 17:05:04.501289 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.497993    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 23 17:05:04.501289 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.497995    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 23 17:05:04.501289 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.497998    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 23 17:05:04.501289 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.498000    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 23 17:05:04.501289 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.498003    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 23 17:05:04.501289 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.498006    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 23 17:05:04.501289 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.498008    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 23 17:05:04.501289 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.498019    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 23 17:05:04.501289 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.498022    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 23 17:05:04.501289 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.498026    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 23 17:05:04.501289 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.498029    2577 feature_gate.go:328] unrecognized feature gate: Example
Jul 23 17:05:04.501289 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.498032    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 23 17:05:04.501289 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.498035    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 23 17:05:04.501289 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.498037    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 23 17:05:04.501289 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.498039    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 23 17:05:04.501289 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.498042    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 23 17:05:04.501289 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.498045    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 23 17:05:04.501289 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.498048    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 23 17:05:04.501759 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.498051    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 23 17:05:04.501759 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.498053    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 23 17:05:04.501759 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.498056    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 23 17:05:04.501759 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.498058    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 23 17:05:04.501759 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.498062    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 23 17:05:04.501759 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.498065    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 23 17:05:04.501759 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.498067    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 23 17:05:04.501759 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.498070    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 23 17:05:04.501759 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.498073    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 23 17:05:04.501759 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.498076    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 23 17:05:04.501759 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.498078    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 23 17:05:04.501759 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.498081    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 23 17:05:04.501759 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.498083    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 23 17:05:04.501759 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.498087    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 23 17:05:04.501759 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.498091    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 23 17:05:04.501759 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.498094    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 23 17:05:04.501759 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.498097    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 23 17:05:04.501759 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.498099    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 23 17:05:04.501759 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.498102    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 23 17:05:04.502206 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.498104    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 23 17:05:04.502206 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.498106    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 23 17:05:04.502206 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.498109    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 23 17:05:04.502206 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.498117    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 23 17:05:04.502206 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.498120    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 23 17:05:04.502206 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.498123    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 23 17:05:04.502206 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.498126    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 23 17:05:04.502206 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.498128    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 23 17:05:04.502206 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.498131    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 23 17:05:04.502206 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.498133    2577 feature_gate.go:328] unrecognized feature gate: Example2
Jul 23 17:05:04.502206 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.498136    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 23 17:05:04.502206 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.498139    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 23 17:05:04.502206 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.498142    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 23 17:05:04.502206 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.498150    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 23 17:05:04.502206 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.498152    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 23 17:05:04.502206 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.498157    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 23 17:05:04.502206 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.498160    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 23 17:05:04.502206 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.498165    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 23 17:05:04.502206 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.498167    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 23 17:05:04.502673 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.498170    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 23 17:05:04.502673 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.498173    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 23 17:05:04.502673 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.498176    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 23 17:05:04.502673 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.498178    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 23 17:05:04.502673 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.498181    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 23 17:05:04.502673 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.498184    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 23 17:05:04.502673 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.498186    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 23 17:05:04.502673 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.498189    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 23 17:05:04.502673 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.498192    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 23 17:05:04.502673 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.498194    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 23 17:05:04.502673 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.498197    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 23 17:05:04.502673 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.498199    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 23 17:05:04.502673 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.498202    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 23 17:05:04.502673 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.498204    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 23 17:05:04.502673 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.498207    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 23 17:05:04.502673 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.498209    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 23 17:05:04.502673 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.498211    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 23 17:05:04.502673 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.498215    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 23 17:05:04.502673 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.498217    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 23 17:05:04.502673 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.498219    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 23 17:05:04.503146 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.498222    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 23 17:05:04.503146 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.498224    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 23 17:05:04.503146 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.498227    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 23 17:05:04.503146 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.498230    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 23 17:05:04.503146 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.498232    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 23 17:05:04.503146 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.498235    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 23 17:05:04.503146 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.498237    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 23 17:05:04.503146 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.498240    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 23 17:05:04.503146 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.498243    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 23 17:05:04.503146 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.498245    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 23 17:05:04.503146 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.498249    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 23 17:05:04.503146 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.498265    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 23 17:05:04.503146 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.498268    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 23 17:05:04.503146 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.498272    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 23 17:05:04.503146 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.498274    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 23 17:05:04.503146 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.498277    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 23 17:05:04.503146 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.498279    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 23 17:05:04.503146 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.498282    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 23 17:05:04.503146 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.498285    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 23 17:05:04.503146 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.498288    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 23 17:05:04.503628 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.498290    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 23 17:05:04.503628 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.498293    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 23 17:05:04.503628 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.498373    2577 flags.go:64] FLAG: --address="0.0.0.0"
Jul 23 17:05:04.503628 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.498383    2577 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 23 17:05:04.503628 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.498393    2577 flags.go:64] FLAG: --anonymous-auth="true"
Jul 23 17:05:04.503628 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.498404    2577 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 23 17:05:04.503628 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.498412    2577 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 23 17:05:04.503628 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.498416    2577 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 23 17:05:04.503628 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.498425    2577 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 23 17:05:04.503628 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.498435    2577 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 23 17:05:04.503628 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.498454    2577 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 23 17:05:04.503628 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.498460    2577 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 23 17:05:04.503628 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.498465    2577 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 23 17:05:04.503628 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.498470    2577 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 23 17:05:04.503628 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.498473    2577 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 23 17:05:04.503628 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.498479    2577 flags.go:64] FLAG: --cgroup-root=""
Jul 23 17:05:04.503628 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.498482    2577 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 23 17:05:04.503628 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.498485    2577 flags.go:64] FLAG: --client-ca-file=""
Jul 23 17:05:04.503628 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.498488    2577 flags.go:64] FLAG: --cloud-provider="external"
Jul 23 17:05:04.503628 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.498491    2577 flags.go:64] FLAG: --cluster-dns="[]"
Jul 23 17:05:04.503628 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.498495    2577 flags.go:64] FLAG: --cluster-domain=""
Jul 23 17:05:04.503628 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.498498    2577 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 23 17:05:04.503628 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.498502    2577 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 23 17:05:04.504158 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.498505    2577 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 23 17:05:04.504158 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.498508    2577 flags.go:64] FLAG: --container-log-max-files="5"
Jul 23 17:05:04.504158 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.498512    2577 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 23 17:05:04.504158 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.498515    2577 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 23 17:05:04.504158 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.498518    2577 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 23 17:05:04.504158 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.498521    2577 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 23 17:05:04.504158 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.498524    2577 flags.go:64] FLAG: --contention-profiling="false"
Jul 23 17:05:04.504158 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.498527    2577 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 23 17:05:04.504158 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.498530    2577 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 23 17:05:04.504158 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.498533    2577 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 23 17:05:04.504158 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.498536    2577 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 23 17:05:04.504158 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.498540    2577 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 23 17:05:04.504158 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.498543    2577 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 23 17:05:04.504158 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.498546    2577 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 23 17:05:04.504158 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.498548    2577 flags.go:64] FLAG: --enable-load-reader="false"
Jul 23 17:05:04.504158 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.498551    2577 flags.go:64] FLAG: --enable-server="true"
Jul 23 17:05:04.504158 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.498554    2577 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 23 17:05:04.504158 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.498562    2577 flags.go:64] FLAG: --event-burst="100"
Jul 23 17:05:04.504158 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.498565    2577 flags.go:64] FLAG: --event-qps="50"
Jul 23 17:05:04.504158 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.498567    2577 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 23 17:05:04.504158 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.498576    2577 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 23 17:05:04.504158 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.498579    2577 flags.go:64] FLAG: --eviction-hard=""
Jul 23 17:05:04.504158 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.498582    2577 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 23 17:05:04.504158 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.498585    2577 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 23 17:05:04.504777 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.498589    2577 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 23 17:05:04.504777 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.498592    2577 flags.go:64] FLAG: --eviction-soft=""
Jul 23 17:05:04.504777 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.498595    2577 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 23 17:05:04.504777 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.498598    2577 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 23 17:05:04.504777 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.498601    2577 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 23 17:05:04.504777 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.498605    2577 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 23 17:05:04.504777 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.498607    2577 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 23 17:05:04.504777 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.498610    2577 flags.go:64] FLAG: --fail-swap-on="true"
Jul 23 17:05:04.504777 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.498613    2577 flags.go:64] FLAG: --feature-gates=""
Jul 23 17:05:04.504777 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.498617    2577 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 23 17:05:04.504777 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.498620    2577 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 23 17:05:04.504777 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.498623    2577 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 23 17:05:04.504777 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.498626    2577 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 23 17:05:04.504777 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.498629    2577 flags.go:64] FLAG: --healthz-port="10248"
Jul 23 17:05:04.504777 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.498631    2577 flags.go:64] FLAG: --help="false"
Jul 23 17:05:04.504777 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.498634    2577 flags.go:64] FLAG: --hostname-override="ip-10-0-129-216.ec2.internal"
Jul 23 17:05:04.504777 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.498637    2577 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 23 17:05:04.504777 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.498640    2577 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 23 17:05:04.504777 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.498643    2577 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 23 17:05:04.504777 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.498646    2577 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 23 17:05:04.504777 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.498650    2577 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 23 17:05:04.504777 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.498652    2577 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 23 17:05:04.504777 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.498655    2577 flags.go:64] FLAG: --image-service-endpoint=""
Jul 23 17:05:04.505408 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.498658    2577 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 23 17:05:04.505408 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.498661    2577 flags.go:64] FLAG: --kube-api-burst="100"
Jul 23 17:05:04.505408 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.498664    2577 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 23 17:05:04.505408 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.498667    2577 flags.go:64] FLAG: --kube-api-qps="50"
Jul 23 17:05:04.505408 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.498669    2577 flags.go:64] FLAG: --kube-reserved=""
Jul 23 17:05:04.505408 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.498672    2577 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 23 17:05:04.505408 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.498681    2577 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 23 17:05:04.505408 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.498684    2577 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 23 17:05:04.505408 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.498687    2577 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 23 17:05:04.505408 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.498691    2577 flags.go:64] FLAG: --lock-file=""
Jul 23 17:05:04.505408 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.498694    2577 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 23 17:05:04.505408 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.498697    2577 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 23 17:05:04.505408 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.498700    2577 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 23 17:05:04.505408 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.498705    2577 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 23 17:05:04.505408 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.498708    2577 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 23 17:05:04.505408 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.498711    2577 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 23 17:05:04.505408 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.498714    2577 flags.go:64] FLAG: --logging-format="text"
Jul 23 17:05:04.505408 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.498716    2577 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 23 17:05:04.505408 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.498720    2577 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 23 17:05:04.505408 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.498723    2577 flags.go:64] FLAG: --manifest-url=""
Jul 23 17:05:04.505408 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.498726    2577 flags.go:64] FLAG: --manifest-url-header=""
Jul 23 17:05:04.505408 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.498730    2577 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 23 17:05:04.505408 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.498732    2577 flags.go:64] FLAG: --max-open-files="1000000"
Jul 23 17:05:04.505408 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.498736    2577 flags.go:64] FLAG: --max-pods="110"
Jul 23 17:05:04.505408 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.498739    2577 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 23 17:05:04.506016 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.498742    2577 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 23 17:05:04.506016 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.498746    2577 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 23 17:05:04.506016 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.498748    2577 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 23 17:05:04.506016 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.498751    2577 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 23 17:05:04.506016 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.498754    2577 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 23 17:05:04.506016 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.498757    2577 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 23 17:05:04.506016 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.498764    2577 flags.go:64] FLAG: --node-status-max-images="50"
Jul 23 17:05:04.506016 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.498767    2577 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 23 17:05:04.506016 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.498770    2577 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 23 17:05:04.506016 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.498773    2577 flags.go:64] FLAG: --pod-cidr=""
Jul 23 17:05:04.506016 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.498776    2577 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4017e99d76d7b91598d3a3430cd0eba6eda8b33653addd3838fdfe3cca69c5a0"
Jul 23 17:05:04.506016 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.498781    2577 flags.go:64] FLAG: --pod-manifest-path=""
Jul 23 17:05:04.506016 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.498783    2577 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 23 17:05:04.506016 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.498786    2577 flags.go:64] FLAG: --pods-per-core="0"
Jul 23 17:05:04.506016 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.498790    2577 flags.go:64] FLAG: --port="10250"
Jul 23 17:05:04.506016 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.498793    2577 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 23 17:05:04.506016 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.498795    2577 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0598fab4c7d454318"
Jul 23 17:05:04.506016 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.498800    2577 flags.go:64] FLAG: --qos-reserved=""
Jul 23 17:05:04.506016 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.498803    2577 flags.go:64] FLAG: --read-only-port="10255"
Jul 23 17:05:04.506016 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.498805    2577 flags.go:64] FLAG: --register-node="true"
Jul 23 17:05:04.506016 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.498809    2577 flags.go:64] FLAG: --register-with-taints=""
Jul 23 17:05:04.506016 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.498812    2577 flags.go:64] FLAG: --registry-burst="10"
Jul 23 17:05:04.506016 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.498815    2577 flags.go:64] FLAG: --registry-qps="5"
Jul 23 17:05:04.506016 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.498818    2577 flags.go:64] FLAG: --reserved-cpus=""
Jul 23 17:05:04.506604 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.498821    2577 flags.go:64] FLAG: --reserved-memory=""
Jul 23 17:05:04.506604 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.498824    2577 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 23 17:05:04.506604 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.498827    2577 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 23 17:05:04.506604 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.498830    2577 flags.go:64] FLAG: --rotate-certificates="false"
Jul 23 17:05:04.506604 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.498833    2577 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 23 17:05:04.506604 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.498836    2577 flags.go:64] FLAG: --runonce="false"
Jul 23 17:05:04.506604 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.498839    2577 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 23 17:05:04.506604 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.498842    2577 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 23 17:05:04.506604 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.498845    2577 flags.go:64] FLAG: --seccomp-default="false"
Jul 23 17:05:04.506604 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.498848    2577 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 23 17:05:04.506604 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.498851    2577 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 23 17:05:04.506604 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.498854    2577 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 23 17:05:04.506604 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.498857    2577 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 23 17:05:04.506604 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.498860    2577 flags.go:64] FLAG: --storage-driver-password="root"
Jul 23 17:05:04.506604 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.498863    2577 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 23 17:05:04.506604 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.498865    2577 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 23 17:05:04.506604 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.498868    2577 flags.go:64] FLAG: --storage-driver-user="root"
Jul 23 17:05:04.506604 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.498871    2577 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 23 17:05:04.506604 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.498874    2577 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 23 17:05:04.506604 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.498877    2577 flags.go:64] FLAG: --system-cgroups=""
Jul 23 17:05:04.506604 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.498880    2577 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 23 17:05:04.506604 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.498885    2577 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 23 17:05:04.506604 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.498888    2577 flags.go:64] FLAG: --tls-cert-file=""
Jul 23 17:05:04.506604 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.498891    2577 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 23 17:05:04.506604 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.498898    2577 flags.go:64] FLAG: --tls-min-version=""
Jul 23 17:05:04.507203 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.498902    2577 flags.go:64] FLAG: --tls-private-key-file=""
Jul 23 17:05:04.507203 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.498905    2577 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 23 17:05:04.507203 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.498909    2577 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 23 17:05:04.507203 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.498912    2577 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 23 17:05:04.507203 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.498915    2577 flags.go:64] FLAG: --v="2"
Jul 23 17:05:04.507203 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.498919    2577 flags.go:64] FLAG: --version="false"
Jul 23 17:05:04.507203 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.498924    2577 flags.go:64] FLAG: --vmodule=""
Jul 23 17:05:04.507203 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.498927    2577 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 23 17:05:04.507203 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.498930    2577 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 23 17:05:04.507203 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.499067    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 23 17:05:04.507203 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.499071    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 23 17:05:04.507203 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.499075    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 23 17:05:04.507203 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.499077    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 23 17:05:04.507203 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.499080    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 23 17:05:04.507203 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.499083    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 23 17:05:04.507203 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.499086    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 23 17:05:04.507203 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.499088    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 23 17:05:04.507203 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.499092    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 23 17:05:04.507203 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.499096    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 23 17:05:04.507203 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.499099    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 23 17:05:04.507203 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.499101    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 23 17:05:04.507203 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.499104    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 23 17:05:04.507751 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.499107    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 23 17:05:04.507751 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.499109    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 23 17:05:04.507751 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.499112    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 23 17:05:04.507751 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.499114    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 23 17:05:04.507751 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.499117    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 23 17:05:04.507751 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.499119    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 23 17:05:04.507751 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.499122    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 23 17:05:04.507751 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.499124    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 23 17:05:04.507751 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.499127    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 23 17:05:04.507751 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.499130    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 23 17:05:04.507751 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.499134    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 23 17:05:04.507751 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.499137    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 23 17:05:04.507751 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.499141    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 23 17:05:04.507751 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.499143    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 23 17:05:04.507751 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.499146    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 23 17:05:04.507751 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.499148    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 23 17:05:04.507751 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.499151    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 23 17:05:04.507751 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.499153    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 23 17:05:04.507751 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.499156    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 23 17:05:04.507751 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.499158    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 23 17:05:04.508297 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.499161    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 23 17:05:04.508297 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.499163    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 23 17:05:04.508297 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.499166    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 23 17:05:04.508297 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.499168    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 23 17:05:04.508297 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.499171    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 23 17:05:04.508297 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.499173    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 23 17:05:04.508297 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.499176    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 23 17:05:04.508297 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.499178    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 23 17:05:04.508297 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.499181    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 23 17:05:04.508297 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.499184    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 23 17:05:04.508297 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.499186    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 23 17:05:04.508297 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.499189    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 23 17:05:04.508297 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.499191    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 23 17:05:04.508297 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.499194    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 23 17:05:04.508297 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.499196    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 23 17:05:04.508297 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.499199    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 23 17:05:04.508297 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.499201    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 23 17:05:04.508297 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.499204    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 23 17:05:04.508297 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.499206    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 23 17:05:04.508297 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.499209    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 23 17:05:04.508778 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.499211    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 23 17:05:04.508778 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.499214    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 23 17:05:04.508778 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.499222    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 23 17:05:04.508778 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.499225    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 23 17:05:04.508778 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.499230    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 23 17:05:04.508778 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.499233    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 23 17:05:04.508778 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.499236    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 23 17:05:04.508778 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.499274    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 23 17:05:04.508778 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.499279    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 23 17:05:04.508778 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.499282    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 23 17:05:04.508778 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.499285    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 23 17:05:04.508778 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.499288    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 23 17:05:04.508778 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.499291    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 23 17:05:04.508778 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.499293    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 23 17:05:04.508778 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.499296    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 23 17:05:04.508778 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.499299    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 23 17:05:04.508778 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.499301    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 23 17:05:04.508778 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.499304    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 23 17:05:04.508778 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.499307    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 23 17:05:04.509230 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.499309    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 23 17:05:04.509230 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.499312    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 23 17:05:04.509230 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.499314    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 23 17:05:04.509230 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.499317    2577 feature_gate.go:328] unrecognized feature gate: Example
Jul 23 17:05:04.509230 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.499319    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 23 17:05:04.509230 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.499322    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 23 17:05:04.509230 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.499325    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 23 17:05:04.509230 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.499327    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 23 17:05:04.509230 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.499329    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 23 17:05:04.509230 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.499332    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 23 17:05:04.509230 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.499334    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 23 17:05:04.509230 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.499337    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 23 17:05:04.509230 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.499339    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 23 17:05:04.509230 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.499342    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 23 17:05:04.509230 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.499344    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 23 17:05:04.509230 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.499348    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 23 17:05:04.509230 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.499351    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 23 17:05:04.509230 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.499355    2577 feature_gate.go:328] unrecognized feature gate: Example2
Jul 23 17:05:04.509230 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.499358    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 23 17:05:04.509230 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.499360    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 23 17:05:04.509723 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.499363    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 23 17:05:04.509723 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.499365    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 23 17:05:04.509723 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.499368    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 23 17:05:04.509723 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.499371    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 23 17:05:04.509723 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.499373    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 23 17:05:04.509723 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.499377    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 23 17:05:04.509723 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.499381    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 23 17:05:04.509723 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.499999    2577 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 23 17:05:04.509723 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.506461    2577 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 23 17:05:04.509723 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.506476    2577 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 23 17:05:04.509723 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.507210    2577 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 23 17:05:04.509723 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.507221    2577 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 23 17:05:04.509723 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.507318    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 23 17:05:04.510047 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.507326    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 23 17:05:04.510047 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.507330    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 23 17:05:04.510047 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.507333    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 23 17:05:04.510047 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.507337    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 23 17:05:04.510047 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.507340    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 23 17:05:04.510047 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.507343    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 23 17:05:04.510047 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.507346    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 23 17:05:04.510047 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.507348    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 23 17:05:04.510047 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.507351    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 23 17:05:04.510047 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.507354    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 23 17:05:04.510047 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.507357    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 23 17:05:04.510047 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.507360    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 23 17:05:04.510047 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.507363    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 23 17:05:04.510047 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.507365    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 23 17:05:04.510047 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.507368    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 23 17:05:04.510047 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.507371    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 23 17:05:04.510047 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.507373    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 23 17:05:04.510047 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.507376    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 23 17:05:04.510047 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.507378    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 23 17:05:04.510047 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.507381    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 23 17:05:04.510571 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.507383    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 23 17:05:04.510571 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.507386    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 23 17:05:04.510571 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.507388    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 23 17:05:04.510571 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.507391    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 23 17:05:04.510571 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.507393    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 23 17:05:04.510571 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.507396    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 23 17:05:04.510571 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.507398    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 23 17:05:04.510571 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.507401    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 23 17:05:04.510571 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.507405    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 23 17:05:04.510571 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.507408    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 23 17:05:04.510571 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.507411    2577 feature_gate.go:328] unrecognized feature gate: Example2
Jul 23 17:05:04.510571 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.507413    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 23 17:05:04.510571 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.507416    2577 feature_gate.go:328] unrecognized feature gate: Example
Jul 23 17:05:04.510571 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.507418    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 23 17:05:04.510571 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.507421    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 23 17:05:04.510571 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.507423    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 23 17:05:04.510571 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.507425    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 23 17:05:04.510571 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.507430    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 23 17:05:04.510571 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.507433    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 23 17:05:04.511022 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.507436    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 23 17:05:04.511022 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.507439    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 23 17:05:04.511022 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.507442    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 23 17:05:04.511022 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.507445    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 23 17:05:04.511022 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.507448    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 23 17:05:04.511022 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.507450    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 23 17:05:04.511022 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.507453    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 23 17:05:04.511022 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.507455    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 23 17:05:04.511022 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.507458    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 23 17:05:04.511022 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.507461    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 23 17:05:04.511022 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.507463    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 23 17:05:04.511022 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.507465    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 23 17:05:04.511022 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.507468    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 23 17:05:04.511022 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.507470    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 23 17:05:04.511022 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.507473    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 23 17:05:04.511022 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.507475    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 23 17:05:04.511022 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.507478    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 23 17:05:04.511022 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.507480    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 23 17:05:04.511022 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.507483    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 23 17:05:04.511022 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.507485    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 23 17:05:04.511506 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.507488    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 23 17:05:04.511506 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.507491    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 23 17:05:04.511506 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.507494    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 23 17:05:04.511506 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.507497    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 23 17:05:04.511506 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.507500    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 23 17:05:04.511506 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.507503    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 23 17:05:04.511506 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.507505    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 23 17:05:04.511506 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.507508    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 23 17:05:04.511506 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.507511    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 23 17:05:04.511506 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.507513    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 23 17:05:04.511506 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.507516    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 23 17:05:04.511506 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.507518    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 23 17:05:04.511506 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.507521    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 23 17:05:04.511506 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.507523    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 23 17:05:04.511506 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.507526    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 23 17:05:04.511506 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.507528    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 23 17:05:04.511506 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.507530    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 23 17:05:04.511506 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.507533    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 23 17:05:04.511506 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.507535    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 23 17:05:04.511506 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.507538    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 23 17:05:04.512016 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.507540    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 23 17:05:04.512016 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.507543    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 23 17:05:04.512016 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.507545    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 23 17:05:04.512016 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.507548    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 23 17:05:04.512016 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.507551    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 23 17:05:04.512016 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.507554    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 23 17:05:04.512016 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.507556    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 23 17:05:04.512016 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.507559    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 23 17:05:04.512016 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.507561    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 23 17:05:04.512016 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.507564    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 23 17:05:04.512016 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.507567    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 23 17:05:04.512016 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.507569    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 23 17:05:04.512016 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.507572    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 23 17:05:04.512016 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.507574    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 23 17:05:04.512016 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.507576    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 23 17:05:04.512016 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.507579    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 23 17:05:04.512016 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.507582    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 23 17:05:04.512016 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.507584    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 23 17:05:04.512016 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.507588    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 23 17:05:04.512479 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.507594    2577 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 23 17:05:04.512479 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.507703    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 23 17:05:04.512479 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.507708    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 23 17:05:04.512479 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.507711    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 23 17:05:04.512479 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.507714    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 23 17:05:04.512479 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.507717    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 23 17:05:04.512479 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.507721    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 23 17:05:04.512479 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.507724    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 23 17:05:04.512479 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.507728    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 23 17:05:04.512479 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.507731    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 23 17:05:04.512479 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.507734    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 23 17:05:04.512479 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.507736    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 23 17:05:04.512479 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.507739    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 23 17:05:04.512479 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.507742    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 23 17:05:04.512815 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.507744    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 23 17:05:04.512815 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.507747    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 23 17:05:04.512815 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.507749    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 23 17:05:04.512815 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.507751    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 23 17:05:04.512815 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.507754    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 23 17:05:04.512815 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.507757    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 23 17:05:04.512815 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.507760    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 23 17:05:04.512815 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.507762    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 23 17:05:04.512815 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.507765    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 23 17:05:04.512815 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.507768    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 23 17:05:04.512815 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.507770    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 23 17:05:04.512815 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.507773    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 23 17:05:04.512815 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.507775    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 23 17:05:04.512815 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.507778    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 23 17:05:04.512815 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.507780    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 23 17:05:04.512815 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.507785    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 23 17:05:04.512815 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.507787    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 23 17:05:04.512815 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.507789    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 23 17:05:04.512815 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.507792    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 23 17:05:04.513317 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.507795    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 23 17:05:04.513317 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.507797    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 23 17:05:04.513317 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.507801    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 23 17:05:04.513317 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.507803    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 23 17:05:04.513317 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.507806    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 23 17:05:04.513317 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.507809    2577 feature_gate.go:328] unrecognized feature gate: Example2
Jul 23 17:05:04.513317 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.507811    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 23 17:05:04.513317 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.507814    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 23 17:05:04.513317 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.507816    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 23 17:05:04.513317 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.507819    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 23 17:05:04.513317 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.507821    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 23 17:05:04.513317 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.507824    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 23 17:05:04.513317 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.507826    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 23 17:05:04.513317 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.507829    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 23 17:05:04.513317 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.507831    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 23 17:05:04.513317 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.507834    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 23 17:05:04.513317 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.507836    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 23 17:05:04.513317 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.507839    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 23 17:05:04.513317 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.507841    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 23 17:05:04.513317 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.507844    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 23 17:05:04.513802 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.507847    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 23 17:05:04.513802 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.507849    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 23 17:05:04.513802 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.507852    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 23 17:05:04.513802 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.507869    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 23 17:05:04.513802 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.507873    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 23 17:05:04.513802 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.507876    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 23 17:05:04.513802 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.507879    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 23 17:05:04.513802 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.507881    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 23 17:05:04.513802 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.507884    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 23 17:05:04.513802 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.507887    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 23 17:05:04.513802 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.507890    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 23 17:05:04.513802 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.507893    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 23 17:05:04.513802 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.507896    2577 feature_gate.go:328] unrecognized feature gate: Example
Jul 23 17:05:04.513802 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.507898    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 23 17:05:04.513802 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.507901    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 23 17:05:04.513802 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.507904    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 23 17:05:04.513802 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.507907    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 23 17:05:04.513802 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.507909    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 23 17:05:04.513802 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.507912    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 23 17:05:04.513802 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.507915    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 23 17:05:04.514279 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.507917    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 23 17:05:04.514279 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.507920    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 23 17:05:04.514279 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.507922    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 23 17:05:04.514279 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.507924    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 23 17:05:04.514279 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.507927    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 23 17:05:04.514279 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.507929    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 23 17:05:04.514279 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.507932    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 23 17:05:04.514279 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.507934    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 23 17:05:04.514279 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.507937    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 23 17:05:04.514279 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.507939    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 23 17:05:04.514279 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.507942    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 23 17:05:04.514279 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.507945    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 23 17:05:04.514279 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.507947    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 23 17:05:04.514279 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.507950    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 23 17:05:04.514279 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.507952    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 23 17:05:04.514279 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.507955    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 23 17:05:04.514279 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.507957    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 23 17:05:04.514279 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.507960    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 23 17:05:04.514279 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.507962    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 23 17:05:04.514279 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.507965    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 23 17:05:04.514787 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.507967    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 23 17:05:04.514787 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.507970    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 23 17:05:04.514787 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.507981    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 23 17:05:04.514787 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.507985    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 23 17:05:04.514787 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.507988    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 23 17:05:04.514787 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.507990    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 23 17:05:04.514787 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:04.507993    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 23 17:05:04.514787 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.507997    2577 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 23 17:05:04.514787 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.508098    2577 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 23 17:05:04.514787 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.510148    2577 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 23 17:05:04.514787 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.511684    2577 server.go:1019] "Starting client certificate rotation"
Jul 23 17:05:04.514787 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.511778    2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 23 17:05:04.514787 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.512412    2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 23 17:05:04.515167 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.514619    2577 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 23 17:05:04.518939 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.518916    2577 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 23 17:05:04.528965 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.528945    2577 log.go:25] "Validated CRI v1 runtime API"
Jul 23 17:05:04.533028 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.533011    2577 log.go:25] "Validated CRI v1 image API"
Jul 23 17:05:04.533803 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.533790    2577 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 23 17:05:04.539229 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.539206    2577 fs.go:135] Filesystem UUIDs: map[1895ada7-dfcc-4ef0-b44b-c0c821d40974:/dev/nvme0n1p4 51633be8-aacf-4511-be67-c2e580b85b1a:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2]
Jul 23 17:05:04.539229 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.539222    2577 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 23 17:05:04.543277 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.543247    2577 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 23 17:05:04.543486 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.543365    2577 manager.go:216] Machine: {Timestamp:2026-07-23 17:05:04.542871718 +0000 UTC m=+0.313157228 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3075093 MemoryCapacity:33164492800 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec231b0108fed22874027d7663313154 SystemUUID:ec231b01-08fe-d228-7402-7d7663313154 BootID:6b3f7848-31b1-4286-b07d-7b23baa3a942 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:b6:5c:bf:37:11 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:b6:5c:bf:37:11 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:be:4b:bc:90:37:eb Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164492800 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 23 17:05:04.543486 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.543481    2577 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 23 17:05:04.543592 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.543561    2577 manager.go:232] Version: {KernelVersion:5.14.0-570.128.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260715-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 23 17:05:04.546231 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.546203    2577 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 23 17:05:04.546377 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.546231    2577 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-129-216.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 23 17:05:04.546908 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.546898    2577 topology_manager.go:138] "Creating topology manager with none policy"
Jul 23 17:05:04.546941 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.546911    2577 container_manager_linux.go:309] "Creating device plugin manager"
Jul 23 17:05:04.546941 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.546918    2577 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 23 17:05:04.547519 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.547510    2577 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 23 17:05:04.549029 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.549019    2577 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 23 17:05:04.551953 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.551943    2577 state_mem.go:36] "Initialized new in-memory state store"
Jul 23 17:05:04.552104 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.552094    2577 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 23 17:05:04.552144 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.552128    2577 kubelet.go:486] "Attempting to sync node with API server"
Jul 23 17:05:04.552180 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.552145    2577 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 23 17:05:04.552180 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.552159    2577 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 23 17:05:04.552180 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.552168    2577 kubelet.go:398] "Adding apiserver pod source"
Jul 23 17:05:04.552180 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.552176    2577 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 23 17:05:04.552535 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.552522    2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 23 17:05:04.552572 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.552549    2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 23 17:05:04.554711 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.554687    2577 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 23 17:05:04.556684 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.556672    2577 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 23 17:05:04.556743 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.556702    2577 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 23 17:05:04.557965 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.557955    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 23 17:05:04.558001 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.557971    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 23 17:05:04.558001 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.557977    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 23 17:05:04.558001 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.557983    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 23 17:05:04.558001 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.557991    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 23 17:05:04.558001 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.557996    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 23 17:05:04.558156 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.558004    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 23 17:05:04.558156 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.558010    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 23 17:05:04.558156 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.558017    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 23 17:05:04.558156 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.558022    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 23 17:05:04.558156 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.558031    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 23 17:05:04.558156 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.558036    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 23 17:05:04.558672 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.558661    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 23 17:05:04.558672 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.558671    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 23 17:05:04.561596 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.561584    2577 server.go:1268] "Started kubelet"
Jul 23 17:05:04.561754 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.561683    2577 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 23 17:05:04.561817 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.561774    2577 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 23 17:05:04.561875 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.561818    2577 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 23 17:05:04.562428 ip-10-0-129-216 systemd[1]: Started Kubernetes Kubelet.
Jul 23 17:05:04.564882 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.564860    2577 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 23 17:05:04.565437 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.565340    2577 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-129-216.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 23 17:05:04.565696 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:05:04.565664    2577 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-129-216.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 23 17:05:04.565773 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:05:04.565702    2577 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 23 17:05:04.568481 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.568147    2577 server.go:310] "Adding debug handlers to kubelet server"
Jul 23 17:05:04.570242 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:05:04.570212    2577 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 23 17:05:04.570642 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.570617    2577 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-8sdcc"
Jul 23 17:05:04.570849 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:05:04.569912    2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-216.ec2.internal.18c4fa114d2a32a3  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-216.ec2.internal,UID:ip-10-0-129-216.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-129-216.ec2.internal,},FirstTimestamp:2026-07-23 17:05:04.561558179 +0000 UTC m=+0.331843689,LastTimestamp:2026-07-23 17:05:04.561558179 +0000 UTC m=+0.331843689,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-216.ec2.internal,}"
Jul 23 17:05:04.571912 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.571894    2577 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 23 17:05:04.572014 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.571914    2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 23 17:05:04.572082 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:05:04.572057    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-216.ec2.internal\" not found"
Jul 23 17:05:04.572504 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.572489    2577 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 23 17:05:04.572750 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.572500    2577 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 23 17:05:04.572750 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.572726    2577 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 23 17:05:04.572887 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.572856    2577 reconstruct.go:97] "Volume reconstruction finished"
Jul 23 17:05:04.572887 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.572864    2577 reconciler.go:29] "Reconciler: start to sync state"
Jul 23 17:05:04.572987 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.572901    2577 factory.go:55] Registering systemd factory
Jul 23 17:05:04.572987 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.572914    2577 factory.go:223] Registration of the systemd container factory successfully
Jul 23 17:05:04.573138 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.573118    2577 factory.go:153] Registering CRI-O factory
Jul 23 17:05:04.573138 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.573138    2577 factory.go:223] Registration of the crio container factory successfully
Jul 23 17:05:04.573289 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.573206    2577 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 23 17:05:04.573289 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.573235    2577 factory.go:103] Registering Raw factory
Jul 23 17:05:04.573289 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.573266    2577 manager.go:1196] Started watching for new ooms in manager
Jul 23 17:05:04.573633 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.573619    2577 manager.go:349] Starting recovery of all containers
Jul 23 17:05:04.575297 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:05:04.575278    2577 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 23 17:05:04.576367 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:05:04.575531    2577 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-129-216.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 23 17:05:04.580680 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.580657    2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-8sdcc"
Jul 23 17:05:04.584750 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.584736    2577 manager.go:354] Recovery completed
Jul 23 17:05:04.586135 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:05:04.586110    2577 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory
Jul 23 17:05:04.589490 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.589476    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 23 17:05:04.591114 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.591086    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-216.ec2.internal" event="NodeHasSufficientMemory"
Jul 23 17:05:04.591181 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.591127    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-216.ec2.internal" event="NodeHasNoDiskPressure"
Jul 23 17:05:04.591181 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.591137    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-216.ec2.internal" event="NodeHasSufficientPID"
Jul 23 17:05:04.591684 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.591661    2577 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 23 17:05:04.591684 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.591674    2577 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 23 17:05:04.591789 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.591690    2577 state_mem.go:36] "Initialized new in-memory state store"
Jul 23 17:05:04.591789 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.591667    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 23 17:05:04.592417 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.592404    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-216.ec2.internal" event="NodeHasSufficientMemory"
Jul 23 17:05:04.592486 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.592430    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-216.ec2.internal" event="NodeHasNoDiskPressure"
Jul 23 17:05:04.592486 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.592446    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-216.ec2.internal" event="NodeHasSufficientPID"
Jul 23 17:05:04.592859 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.592846    2577 policy_none.go:49] "None policy: Start"
Jul 23 17:05:04.592911 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.592862    2577 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 23 17:05:04.592911 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.592872    2577 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 23 17:05:04.593515 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:05:04.593455    2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-216.ec2.internal.18c4fa114eed29c7  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-216.ec2.internal,UID:ip-10-0-129-216.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-129-216.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-129-216.ec2.internal,},FirstTimestamp:2026-07-23 17:05:04.591112647 +0000 UTC m=+0.361398157,LastTimestamp:2026-07-23 17:05:04.591112647 +0000 UTC m=+0.361398157,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-216.ec2.internal,}"
Jul 23 17:05:04.593995 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.593983    2577 policy_none.go:47] "Start"
Jul 23 17:05:04.633805 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.633786    2577 manager.go:337] "Starting Device Plugin manager"
Jul 23 17:05:04.633894 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:05:04.633818    2577 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 23 17:05:04.633894 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.633828    2577 server.go:87] "Starting device plugin registration server"
Jul 23 17:05:04.634062 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.634049    2577 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 23 17:05:04.634106 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.634064    2577 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 23 17:05:04.634229 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.634212    2577 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 23 17:05:04.634321 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.634305    2577 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 23 17:05:04.634321 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.634317    2577 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 23 17:05:04.634707 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:05:04.634689    2577 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 23 17:05:04.634786 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:05:04.634739    2577 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-129-216.ec2.internal\" not found"
Jul 23 17:05:04.734987 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.734831    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 23 17:05:04.736059 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.736034    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-216.ec2.internal" event="NodeHasSufficientMemory"
Jul 23 17:05:04.736186 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.736068    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-216.ec2.internal" event="NodeHasNoDiskPressure"
Jul 23 17:05:04.736186 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.736080    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-216.ec2.internal" event="NodeHasSufficientPID"
Jul 23 17:05:04.736186 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.736109    2577 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-129-216.ec2.internal"
Jul 23 17:05:04.742645 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.742625    2577 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-129-216.ec2.internal"
Jul 23 17:05:04.742719 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:05:04.742649    2577 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-129-216.ec2.internal\": node \"ip-10-0-129-216.ec2.internal\" not found"
Jul 23 17:05:04.757891 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.757826    2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 23 17:05:04.758285 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:05:04.758266    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-216.ec2.internal\" not found"
Jul 23 17:05:04.759150 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.759123    2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 23 17:05:04.759237 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.759162    2577 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 23 17:05:04.759237 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.759182    2577 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 23 17:05:04.759237 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:05:04.759217    2577 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 23 17:05:04.761534 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.761514    2577 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 23 17:05:04.858715 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:05:04.858679    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-216.ec2.internal\" not found"
Jul 23 17:05:04.859881 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.859862    2577 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-216.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-129-216.ec2.internal"]
Jul 23 17:05:04.859946 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.859934    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 23 17:05:04.863526 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.863510    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-216.ec2.internal" event="NodeHasSufficientMemory"
Jul 23 17:05:04.863648 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.863539    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-216.ec2.internal" event="NodeHasNoDiskPressure"
Jul 23 17:05:04.863648 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.863551    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-216.ec2.internal" event="NodeHasSufficientPID"
Jul 23 17:05:04.864928 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.864915    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 23 17:05:04.865051 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.865036    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-216.ec2.internal"
Jul 23 17:05:04.865093 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.865068    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 23 17:05:04.868134 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.868118    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-216.ec2.internal" event="NodeHasSufficientMemory"
Jul 23 17:05:04.868134 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.868127    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-216.ec2.internal" event="NodeHasSufficientMemory"
Jul 23 17:05:04.868302 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.868146    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-216.ec2.internal" event="NodeHasNoDiskPressure"
Jul 23 17:05:04.868302 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.868157    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-216.ec2.internal" event="NodeHasSufficientPID"
Jul 23 17:05:04.868302 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.868147    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-216.ec2.internal" event="NodeHasNoDiskPressure"
Jul 23 17:05:04.868302 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.868229    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-216.ec2.internal" event="NodeHasSufficientPID"
Jul 23 17:05:04.869412 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.869396    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-216.ec2.internal"
Jul 23 17:05:04.869470 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.869423    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 23 17:05:04.870237 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.870219    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-216.ec2.internal" event="NodeHasSufficientMemory"
Jul 23 17:05:04.870289 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.870253    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-216.ec2.internal" event="NodeHasNoDiskPressure"
Jul 23 17:05:04.870289 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.870283    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-216.ec2.internal" event="NodeHasSufficientPID"
Jul 23 17:05:04.874862 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.874841    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/28346eafc8fa850eaa3541eee81eaa75-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-216.ec2.internal\" (UID: \"28346eafc8fa850eaa3541eee81eaa75\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-216.ec2.internal"
Jul 23 17:05:04.874945 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.874878    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/28346eafc8fa850eaa3541eee81eaa75-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-216.ec2.internal\" (UID: \"28346eafc8fa850eaa3541eee81eaa75\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-216.ec2.internal"
Jul 23 17:05:04.874945 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.874915    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7bd1d5c1fbf257c4a5dcb5c0b12f7f75-config\") pod \"kube-apiserver-proxy-ip-10-0-129-216.ec2.internal\" (UID: \"7bd1d5c1fbf257c4a5dcb5c0b12f7f75\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-216.ec2.internal"
Jul 23 17:05:04.887912 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:05:04.887883    2577 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-216.ec2.internal\" not found" node="ip-10-0-129-216.ec2.internal"
Jul 23 17:05:04.892682 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:05:04.892667    2577 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-216.ec2.internal\" not found" node="ip-10-0-129-216.ec2.internal"
Jul 23 17:05:04.960731 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:05:04.960702    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-216.ec2.internal\" not found"
Jul 23 17:05:04.975074 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.975046    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7bd1d5c1fbf257c4a5dcb5c0b12f7f75-config\") pod \"kube-apiserver-proxy-ip-10-0-129-216.ec2.internal\" (UID: \"7bd1d5c1fbf257c4a5dcb5c0b12f7f75\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-216.ec2.internal"
Jul 23 17:05:04.975142 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.975090    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/28346eafc8fa850eaa3541eee81eaa75-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-216.ec2.internal\" (UID: \"28346eafc8fa850eaa3541eee81eaa75\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-216.ec2.internal"
Jul 23 17:05:04.975142 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.975108    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/28346eafc8fa850eaa3541eee81eaa75-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-216.ec2.internal\" (UID: \"28346eafc8fa850eaa3541eee81eaa75\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-216.ec2.internal"
Jul 23 17:05:04.975142 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.975128    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7bd1d5c1fbf257c4a5dcb5c0b12f7f75-config\") pod \"kube-apiserver-proxy-ip-10-0-129-216.ec2.internal\" (UID: \"7bd1d5c1fbf257c4a5dcb5c0b12f7f75\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-216.ec2.internal"
Jul 23 17:05:04.975243 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.975151    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/28346eafc8fa850eaa3541eee81eaa75-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-216.ec2.internal\" (UID: \"28346eafc8fa850eaa3541eee81eaa75\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-216.ec2.internal"
Jul 23 17:05:04.975243 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:04.975154    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/28346eafc8fa850eaa3541eee81eaa75-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-216.ec2.internal\" (UID: \"28346eafc8fa850eaa3541eee81eaa75\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-216.ec2.internal"
Jul 23 17:05:05.061724 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:05:05.061661    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-216.ec2.internal\" not found"
Jul 23 17:05:05.163263 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:05:05.163232    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-216.ec2.internal\" not found"
Jul 23 17:05:05.189323 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:05.189291    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-216.ec2.internal"
Jul 23 17:05:05.196976 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:05.196952    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-216.ec2.internal"
Jul 23 17:05:05.264607 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:05:05.264574    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-216.ec2.internal\" not found"
Jul 23 17:05:05.365518 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:05:05.365454    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-216.ec2.internal\" not found"
Jul 23 17:05:05.466914 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:05:05.466878    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-216.ec2.internal\" not found"
Jul 23 17:05:05.510643 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:05.510606    2577 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 23 17:05:05.511052 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:05.510767    2577 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 23 17:05:05.567308 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:05:05.567285    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-216.ec2.internal\" not found"
Jul 23 17:05:05.589303 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:05.589269    2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-22 17:00:04 +0000 UTC" deadline="2028-02-24 12:40:22.707108809 +0000 UTC"
Jul 23 17:05:05.589303 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:05.589298    2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13939h35m17.117814269s"
Jul 23 17:05:05.668132 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:05:05.668071    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-216.ec2.internal\" not found"
Jul 23 17:05:05.769135 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:05:05.769111    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-216.ec2.internal\" not found"
Jul 23 17:05:05.833554 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:05.833493    2577 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 23 17:05:05.869226 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:05:05.869197    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-216.ec2.internal\" not found"
Jul 23 17:05:05.884033 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:05.883997    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bd1d5c1fbf257c4a5dcb5c0b12f7f75.slice/crio-38d0bb6f5de6697f84b4329230e2823bcaeb19960c94283042782fdee2b520ff WatchSource:0}: Error finding container 38d0bb6f5de6697f84b4329230e2823bcaeb19960c94283042782fdee2b520ff: Status 404 returned error can't find the container with id 38d0bb6f5de6697f84b4329230e2823bcaeb19960c94283042782fdee2b520ff
Jul 23 17:05:05.888128 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:05.888112    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 23 17:05:05.948999 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:05.948981    2577 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 23 17:05:05.969835 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:05:05.969810    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-216.ec2.internal\" not found"
Jul 23 17:05:06.041683 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.041657    2577 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 23 17:05:06.072092 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.072070    2577 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-216.ec2.internal"
Jul 23 17:05:06.085869 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.085852    2577 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 23 17:05:06.086938 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.086925    2577 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-216.ec2.internal"
Jul 23 17:05:06.097763 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.097746    2577 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 23 17:05:06.555664 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.555633    2577 apiserver.go:52] "Watching apiserver"
Jul 23 17:05:06.568298 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.568273    2577 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 23 17:05:06.568780 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.568736    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-216.ec2.internal","openshift-multus/multus-6hk44","openshift-multus/multus-additional-cni-plugins-b797v","openshift-network-operator/iptables-alerter-ft9nr","kube-system/konnectivity-agent-pc8xz","openshift-cluster-node-tuning-operator/tuned-vj6t5","openshift-dns/node-resolver-stw6g","openshift-multus/network-metrics-daemon-mjfnh","openshift-network-diagnostics/network-check-target-5c6b5","openshift-ovn-kubernetes/ovnkube-node-2s9nz","kube-system/global-pull-secret-syncer-5g97l","kube-system/kube-apiserver-proxy-ip-10-0-129-216.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p4twn","openshift-image-registry/node-ca-znmk5"]
Jul 23 17:05:06.570308 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.570284    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-stw6g"
Jul 23 17:05:06.572072 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.572051    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p4twn"
Jul 23 17:05:06.572923 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.572904    2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 23 17:05:06.573458 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.573436    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-b797v"
Jul 23 17:05:06.574817 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.574794    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-ft9nr"
Jul 23 17:05:06.575081 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.575061    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-8q4fd\""
Jul 23 17:05:06.575397 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.575374    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 23 17:05:06.575484 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.575406    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 23 17:05:06.576361 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.576147    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-pc8xz"
Jul 23 17:05:06.577456 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.577435    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-vj6t5"
Jul 23 17:05:06.578969 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.578951    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5c6b5"
Jul 23 17:05:06.579066 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:05:06.579030    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5c6b5" podUID="2090b02e-14f1-4224-8691-8fa4ecf7d53b"
Jul 23 17:05:06.580389 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.580317    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-znmk5"
Jul 23 17:05:06.581900 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.581880    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-6hk44"
Jul 23 17:05:06.582534 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.582514    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 23 17:05:06.583115 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.582969    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 23 17:05:06.583115 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.582992    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-9nkvn\""
Jul 23 17:05:06.583675 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.583616    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-2s9nz"
Jul 23 17:05:06.583946 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.583927    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 23 17:05:06.584462 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.584441    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 23 17:05:06.584555 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.584513    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 23 17:05:06.584649 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.584453    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 23 17:05:06.584735 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.584712    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 23 17:05:06.585003 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.584978    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 23 17:05:06.585163 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.585143    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 23 17:05:06.585358 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.585318    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-zmzlg\""
Jul 23 17:05:06.585450 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.585320    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 23 17:05:06.585450 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.585332    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/71b6eaa7-70df-4f01-b0e9-6be6043acbc4-sys-fs\") pod \"aws-ebs-csi-driver-node-p4twn\" (UID: \"71b6eaa7-70df-4f01-b0e9-6be6043acbc4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p4twn"
Jul 23 17:05:06.585450 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.585433    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 23 17:05:06.585610 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.585456    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/5c929500-877b-43d7-9710-0ea27db45055-hosts-file\") pod \"node-resolver-stw6g\" (UID: \"5c929500-877b-43d7-9710-0ea27db45055\") " pod="openshift-dns/node-resolver-stw6g"
Jul 23 17:05:06.585610 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.585484    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/71b6eaa7-70df-4f01-b0e9-6be6043acbc4-socket-dir\") pod \"aws-ebs-csi-driver-node-p4twn\" (UID: \"71b6eaa7-70df-4f01-b0e9-6be6043acbc4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p4twn"
Jul 23 17:05:06.585610 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.585510    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/65acd752-a782-40a8-82b9-3891d7349f33-system-cni-dir\") pod \"multus-additional-cni-plugins-b797v\" (UID: \"65acd752-a782-40a8-82b9-3891d7349f33\") " pod="openshift-multus/multus-additional-cni-plugins-b797v"
Jul 23 17:05:06.585610 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.585336    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 23 17:05:06.585610 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.585538    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/65acd752-a782-40a8-82b9-3891d7349f33-os-release\") pod \"multus-additional-cni-plugins-b797v\" (UID: \"65acd752-a782-40a8-82b9-3891d7349f33\") " pod="openshift-multus/multus-additional-cni-plugins-b797v"
Jul 23 17:05:06.585610 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.585570    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/65acd752-a782-40a8-82b9-3891d7349f33-tuning-conf-dir\") pod \"multus-additional-cni-plugins-b797v\" (UID: \"65acd752-a782-40a8-82b9-3891d7349f33\") " pod="openshift-multus/multus-additional-cni-plugins-b797v"
Jul 23 17:05:06.585919 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.585627    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/65acd752-a782-40a8-82b9-3891d7349f33-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-b797v\" (UID: \"65acd752-a782-40a8-82b9-3891d7349f33\") " pod="openshift-multus/multus-additional-cni-plugins-b797v"
Jul 23 17:05:06.585919 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.585674    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twxbm\" (UniqueName: \"kubernetes.io/projected/5c929500-877b-43d7-9710-0ea27db45055-kube-api-access-twxbm\") pod \"node-resolver-stw6g\" (UID: \"5c929500-877b-43d7-9710-0ea27db45055\") " pod="openshift-dns/node-resolver-stw6g"
Jul 23 17:05:06.585919 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.585719    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/71b6eaa7-70df-4f01-b0e9-6be6043acbc4-registration-dir\") pod \"aws-ebs-csi-driver-node-p4twn\" (UID: \"71b6eaa7-70df-4f01-b0e9-6be6043acbc4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p4twn"
Jul 23 17:05:06.585919 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.585758    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/71b6eaa7-70df-4f01-b0e9-6be6043acbc4-etc-selinux\") pod \"aws-ebs-csi-driver-node-p4twn\" (UID: \"71b6eaa7-70df-4f01-b0e9-6be6043acbc4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p4twn"
Jul 23 17:05:06.585919 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.585773    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 23 17:05:06.585919 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.585786    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/65acd752-a782-40a8-82b9-3891d7349f33-cnibin\") pod \"multus-additional-cni-plugins-b797v\" (UID: \"65acd752-a782-40a8-82b9-3891d7349f33\") " pod="openshift-multus/multus-additional-cni-plugins-b797v"
Jul 23 17:05:06.585919 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.585799    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 23 17:05:06.585919 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.585840    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c929500-877b-43d7-9710-0ea27db45055-tmp-dir\") pod \"node-resolver-stw6g\" (UID: \"5c929500-877b-43d7-9710-0ea27db45055\") " pod="openshift-dns/node-resolver-stw6g"
Jul 23 17:05:06.585919 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.585865    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/71b6eaa7-70df-4f01-b0e9-6be6043acbc4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-p4twn\" (UID: \"71b6eaa7-70df-4f01-b0e9-6be6043acbc4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p4twn"
Jul 23 17:05:06.585919 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.585890    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/71b6eaa7-70df-4f01-b0e9-6be6043acbc4-device-dir\") pod \"aws-ebs-csi-driver-node-p4twn\" (UID: \"71b6eaa7-70df-4f01-b0e9-6be6043acbc4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p4twn"
Jul 23 17:05:06.585919 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.585903    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-ggqzl\""
Jul 23 17:05:06.585919 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.585919    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zs94q\" (UniqueName: \"kubernetes.io/projected/71b6eaa7-70df-4f01-b0e9-6be6043acbc4-kube-api-access-zs94q\") pod \"aws-ebs-csi-driver-node-p4twn\" (UID: \"71b6eaa7-70df-4f01-b0e9-6be6043acbc4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p4twn"
Jul 23 17:05:06.586514 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.585971    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/65acd752-a782-40a8-82b9-3891d7349f33-cni-binary-copy\") pod \"multus-additional-cni-plugins-b797v\" (UID: \"65acd752-a782-40a8-82b9-3891d7349f33\") " pod="openshift-multus/multus-additional-cni-plugins-b797v"
Jul 23 17:05:06.586514 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.585998    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/65acd752-a782-40a8-82b9-3891d7349f33-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-b797v\" (UID: \"65acd752-a782-40a8-82b9-3891d7349f33\") " pod="openshift-multus/multus-additional-cni-plugins-b797v"
Jul 23 17:05:06.586514 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.586025    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4dcq\" (UniqueName: \"kubernetes.io/projected/65acd752-a782-40a8-82b9-3891d7349f33-kube-api-access-w4dcq\") pod \"multus-additional-cni-plugins-b797v\" (UID: \"65acd752-a782-40a8-82b9-3891d7349f33\") " pod="openshift-multus/multus-additional-cni-plugins-b797v"
Jul 23 17:05:06.586514 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.586053    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 23 17:05:06.586514 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.586117    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 23 17:05:06.586514 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.586140    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 23 17:05:06.586514 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.586513    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-c5cv9\""
Jul 23 17:05:06.586873 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.586532    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-wdkhj\""
Jul 23 17:05:06.586873 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.586710    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 23 17:05:06.586873 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.586758    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-7rq6j\""
Jul 23 17:05:06.586873 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.586802    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-4q96l\""
Jul 23 17:05:06.587079 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.587019    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5g97l"
Jul 23 17:05:06.587138 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:05:06.587091    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5g97l" podUID="aeb10e6f-ad01-4a83-8090-049d9e919696"
Jul 23 17:05:06.587314 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.587282    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 23 17:05:06.587416 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.587322    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 23 17:05:06.587732 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.587580    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 23 17:05:06.587732 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.587582    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-r689f\""
Jul 23 17:05:06.588243 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.588219    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mjfnh"
Jul 23 17:05:06.588350 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:05:06.588279    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mjfnh" podUID="664f5a33-efcf-447f-8e05-a3f6ce7e4f07"
Jul 23 17:05:06.589537 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.589395    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 23 17:05:06.589537 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.589429    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 23 17:05:06.589765 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.589747    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 23 17:05:06.593434 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.593403    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 23 17:05:06.608711 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.608686    2577 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 23 17:05:06.644870 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.644849    2577 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-xnksr"
Jul 23 17:05:06.658736 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.658711    2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-xnksr"
Jul 23 17:05:06.673964 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.673940    2577 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 23 17:05:06.686802 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.686770    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hpn22\" (UniqueName: \"kubernetes.io/projected/2090b02e-14f1-4224-8691-8fa4ecf7d53b-kube-api-access-hpn22\") pod \"network-check-target-5c6b5\" (UID: \"2090b02e-14f1-4224-8691-8fa4ecf7d53b\") " pod="openshift-network-diagnostics/network-check-target-5c6b5"
Jul 23 17:05:06.686885 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.686812    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/71b6eaa7-70df-4f01-b0e9-6be6043acbc4-socket-dir\") pod \"aws-ebs-csi-driver-node-p4twn\" (UID: \"71b6eaa7-70df-4f01-b0e9-6be6043acbc4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p4twn"
Jul 23 17:05:06.686885 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.686832    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/65acd752-a782-40a8-82b9-3891d7349f33-os-release\") pod \"multus-additional-cni-plugins-b797v\" (UID: \"65acd752-a782-40a8-82b9-3891d7349f33\") " pod="openshift-multus/multus-additional-cni-plugins-b797v"
Jul 23 17:05:06.686885 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.686875    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/62550ab4-e316-461d-a09e-679202149bb2-etc-sysconfig\") pod \"tuned-vj6t5\" (UID: \"62550ab4-e316-461d-a09e-679202149bb2\") " pod="openshift-cluster-node-tuning-operator/tuned-vj6t5"
Jul 23 17:05:06.687064 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.686954    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/65acd752-a782-40a8-82b9-3891d7349f33-os-release\") pod \"multus-additional-cni-plugins-b797v\" (UID: \"65acd752-a782-40a8-82b9-3891d7349f33\") " pod="openshift-multus/multus-additional-cni-plugins-b797v"
Jul 23 17:05:06.687064 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.686956    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/62550ab4-e316-461d-a09e-679202149bb2-sys\") pod \"tuned-vj6t5\" (UID: \"62550ab4-e316-461d-a09e-679202149bb2\") " pod="openshift-cluster-node-tuning-operator/tuned-vj6t5"
Jul 23 17:05:06.687064 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.686954    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/71b6eaa7-70df-4f01-b0e9-6be6043acbc4-socket-dir\") pod \"aws-ebs-csi-driver-node-p4twn\" (UID: \"71b6eaa7-70df-4f01-b0e9-6be6043acbc4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p4twn"
Jul 23 17:05:06.687064 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.686993    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a70b8d80-c6a7-4498-ab2d-d1a7d3957ceb-host-var-lib-cni-bin\") pod \"multus-6hk44\" (UID: \"a70b8d80-c6a7-4498-ab2d-d1a7d3957ceb\") " pod="openshift-multus/multus-6hk44"
Jul 23 17:05:06.687064 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.687021    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c5765452-37b1-4153-9096-778d6b33857c-run-openvswitch\") pod \"ovnkube-node-2s9nz\" (UID: \"c5765452-37b1-4153-9096-778d6b33857c\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s9nz"
Jul 23 17:05:06.687064 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.687049    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c5765452-37b1-4153-9096-778d6b33857c-log-socket\") pod \"ovnkube-node-2s9nz\" (UID: \"c5765452-37b1-4153-9096-778d6b33857c\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s9nz"
Jul 23 17:05:06.687375 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.687087    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvl55\" (UniqueName: \"kubernetes.io/projected/c5765452-37b1-4153-9096-778d6b33857c-kube-api-access-rvl55\") pod \"ovnkube-node-2s9nz\" (UID: \"c5765452-37b1-4153-9096-778d6b33857c\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s9nz"
Jul 23 17:05:06.687375 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.687118    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a70b8d80-c6a7-4498-ab2d-d1a7d3957ceb-system-cni-dir\") pod \"multus-6hk44\" (UID: \"a70b8d80-c6a7-4498-ab2d-d1a7d3957ceb\") " pod="openshift-multus/multus-6hk44"
Jul 23 17:05:06.687375 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.687142    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a70b8d80-c6a7-4498-ab2d-d1a7d3957ceb-hostroot\") pod \"multus-6hk44\" (UID: \"a70b8d80-c6a7-4498-ab2d-d1a7d3957ceb\") " pod="openshift-multus/multus-6hk44"
Jul 23 17:05:06.687375 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.687168    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/71b6eaa7-70df-4f01-b0e9-6be6043acbc4-device-dir\") pod \"aws-ebs-csi-driver-node-p4twn\" (UID: \"71b6eaa7-70df-4f01-b0e9-6be6043acbc4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p4twn"
Jul 23 17:05:06.687375 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.687205    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zs94q\" (UniqueName: \"kubernetes.io/projected/71b6eaa7-70df-4f01-b0e9-6be6043acbc4-kube-api-access-zs94q\") pod \"aws-ebs-csi-driver-node-p4twn\" (UID: \"71b6eaa7-70df-4f01-b0e9-6be6043acbc4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p4twn"
Jul 23 17:05:06.687375 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.687255    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/71b6eaa7-70df-4f01-b0e9-6be6043acbc4-device-dir\") pod \"aws-ebs-csi-driver-node-p4twn\" (UID: \"71b6eaa7-70df-4f01-b0e9-6be6043acbc4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p4twn"
Jul 23 17:05:06.687375 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.687314    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/62550ab4-e316-461d-a09e-679202149bb2-etc-tuned\") pod \"tuned-vj6t5\" (UID: \"62550ab4-e316-461d-a09e-679202149bb2\") " pod="openshift-cluster-node-tuning-operator/tuned-vj6t5"
Jul 23 17:05:06.687685 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.687402    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c5765452-37b1-4153-9096-778d6b33857c-host-cni-netd\") pod \"ovnkube-node-2s9nz\" (UID: \"c5765452-37b1-4153-9096-778d6b33857c\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s9nz"
Jul 23 17:05:06.687685 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.687428    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c5765452-37b1-4153-9096-778d6b33857c-ovn-node-metrics-cert\") pod \"ovnkube-node-2s9nz\" (UID: \"c5765452-37b1-4153-9096-778d6b33857c\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s9nz"
Jul 23 17:05:06.687685 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.687453    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/62550ab4-e316-461d-a09e-679202149bb2-etc-kubernetes\") pod \"tuned-vj6t5\" (UID: \"62550ab4-e316-461d-a09e-679202149bb2\") " pod="openshift-cluster-node-tuning-operator/tuned-vj6t5"
Jul 23 17:05:06.687685 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.687478    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/62550ab4-e316-461d-a09e-679202149bb2-etc-sysctl-conf\") pod \"tuned-vj6t5\" (UID: \"62550ab4-e316-461d-a09e-679202149bb2\") " pod="openshift-cluster-node-tuning-operator/tuned-vj6t5"
Jul 23 17:05:06.687685 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.687502    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/62550ab4-e316-461d-a09e-679202149bb2-var-lib-kubelet\") pod \"tuned-vj6t5\" (UID: \"62550ab4-e316-461d-a09e-679202149bb2\") " pod="openshift-cluster-node-tuning-operator/tuned-vj6t5"
Jul 23 17:05:06.687685 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.687528    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a70b8d80-c6a7-4498-ab2d-d1a7d3957ceb-host-var-lib-cni-multus\") pod \"multus-6hk44\" (UID: \"a70b8d80-c6a7-4498-ab2d-d1a7d3957ceb\") " pod="openshift-multus/multus-6hk44"
Jul 23 17:05:06.687685 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.687554    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c5765452-37b1-4153-9096-778d6b33857c-systemd-units\") pod \"ovnkube-node-2s9nz\" (UID: \"c5765452-37b1-4153-9096-778d6b33857c\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s9nz"
Jul 23 17:05:06.687685 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.687581    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/88de4e3c-bd4f-44d7-a9aa-58c2c4d24549-host-slash\") pod \"iptables-alerter-ft9nr\" (UID: \"88de4e3c-bd4f-44d7-a9aa-58c2c4d24549\") " pod="openshift-network-operator/iptables-alerter-ft9nr"
Jul 23 17:05:06.687685 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.687624    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c5765452-37b1-4153-9096-778d6b33857c-var-lib-openvswitch\") pod \"ovnkube-node-2s9nz\" (UID: \"c5765452-37b1-4153-9096-778d6b33857c\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s9nz"
Jul 23 17:05:06.687685 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.687648    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c5765452-37b1-4153-9096-778d6b33857c-env-overrides\") pod \"ovnkube-node-2s9nz\" (UID: \"c5765452-37b1-4153-9096-778d6b33857c\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s9nz"
Jul 23 17:05:06.687685 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.687672    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a70b8d80-c6a7-4498-ab2d-d1a7d3957ceb-multus-socket-dir-parent\") pod \"multus-6hk44\" (UID: \"a70b8d80-c6a7-4498-ab2d-d1a7d3957ceb\") " pod="openshift-multus/multus-6hk44"
Jul 23 17:05:06.688113 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.687701    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twxbm\" (UniqueName: \"kubernetes.io/projected/5c929500-877b-43d7-9710-0ea27db45055-kube-api-access-twxbm\") pod \"node-resolver-stw6g\" (UID: \"5c929500-877b-43d7-9710-0ea27db45055\") " pod="openshift-dns/node-resolver-stw6g"
Jul 23 17:05:06.688113 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.687726    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c5765452-37b1-4153-9096-778d6b33857c-host-run-netns\") pod \"ovnkube-node-2s9nz\" (UID: \"c5765452-37b1-4153-9096-778d6b33857c\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s9nz"
Jul 23 17:05:06.688113 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.687751    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c5765452-37b1-4153-9096-778d6b33857c-run-ovn\") pod \"ovnkube-node-2s9nz\" (UID: \"c5765452-37b1-4153-9096-778d6b33857c\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s9nz"
Jul 23 17:05:06.688113 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.687775    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c5765452-37b1-4153-9096-778d6b33857c-node-log\") pod \"ovnkube-node-2s9nz\" (UID: \"c5765452-37b1-4153-9096-778d6b33857c\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s9nz"
Jul 23 17:05:06.688113 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.687799    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/aeb10e6f-ad01-4a83-8090-049d9e919696-kubelet-config\") pod \"global-pull-secret-syncer-5g97l\" (UID: \"aeb10e6f-ad01-4a83-8090-049d9e919696\") " pod="kube-system/global-pull-secret-syncer-5g97l"
Jul 23 17:05:06.688113 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.687839    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/aeb10e6f-ad01-4a83-8090-049d9e919696-dbus\") pod \"global-pull-secret-syncer-5g97l\" (UID: \"aeb10e6f-ad01-4a83-8090-049d9e919696\") " pod="kube-system/global-pull-secret-syncer-5g97l"
Jul 23 17:05:06.688113 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.687866    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/aeb10e6f-ad01-4a83-8090-049d9e919696-original-pull-secret\") pod \"global-pull-secret-syncer-5g97l\" (UID: \"aeb10e6f-ad01-4a83-8090-049d9e919696\") " pod="kube-system/global-pull-secret-syncer-5g97l"
Jul 23 17:05:06.688113 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.687894    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c929500-877b-43d7-9710-0ea27db45055-tmp-dir\") pod \"node-resolver-stw6g\" (UID: \"5c929500-877b-43d7-9710-0ea27db45055\") " pod="openshift-dns/node-resolver-stw6g"
Jul 23 17:05:06.688113 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.687918    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/62550ab4-e316-461d-a09e-679202149bb2-etc-modprobe-d\") pod \"tuned-vj6t5\" (UID: \"62550ab4-e316-461d-a09e-679202149bb2\") " pod="openshift-cluster-node-tuning-operator/tuned-vj6t5"
Jul 23 17:05:06.688113 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.687942    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/62550ab4-e316-461d-a09e-679202149bb2-host\") pod \"tuned-vj6t5\" (UID: \"62550ab4-e316-461d-a09e-679202149bb2\") " pod="openshift-cluster-node-tuning-operator/tuned-vj6t5"
Jul 23 17:05:06.688113 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.687973    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvc29\" (UniqueName: \"kubernetes.io/projected/62550ab4-e316-461d-a09e-679202149bb2-kube-api-access-nvc29\") pod \"tuned-vj6t5\" (UID: \"62550ab4-e316-461d-a09e-679202149bb2\") " pod="openshift-cluster-node-tuning-operator/tuned-vj6t5"
Jul 23 17:05:06.688113 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.687997    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a70b8d80-c6a7-4498-ab2d-d1a7d3957ceb-cnibin\") pod \"multus-6hk44\" (UID: \"a70b8d80-c6a7-4498-ab2d-d1a7d3957ceb\") " pod="openshift-multus/multus-6hk44"
Jul 23 17:05:06.688113 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.688041    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a70b8d80-c6a7-4498-ab2d-d1a7d3957ceb-host-run-k8s-cni-cncf-io\") pod \"multus-6hk44\" (UID: \"a70b8d80-c6a7-4498-ab2d-d1a7d3957ceb\") " pod="openshift-multus/multus-6hk44"
Jul 23 17:05:06.688113 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.688066    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a70b8d80-c6a7-4498-ab2d-d1a7d3957ceb-multus-daemon-config\") pod \"multus-6hk44\" (UID: \"a70b8d80-c6a7-4498-ab2d-d1a7d3957ceb\") " pod="openshift-multus/multus-6hk44"
Jul 23 17:05:06.688113 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.688092    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c5765452-37b1-4153-9096-778d6b33857c-ovnkube-script-lib\") pod \"ovnkube-node-2s9nz\" (UID: \"c5765452-37b1-4153-9096-778d6b33857c\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s9nz"
Jul 23 17:05:06.688940 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.688130    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a70b8d80-c6a7-4498-ab2d-d1a7d3957ceb-host-run-netns\") pod \"multus-6hk44\" (UID: \"a70b8d80-c6a7-4498-ab2d-d1a7d3957ceb\") " pod="openshift-multus/multus-6hk44"
Jul 23 17:05:06.688940 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.688170    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a70b8d80-c6a7-4498-ab2d-d1a7d3957ceb-multus-conf-dir\") pod \"multus-6hk44\" (UID: \"a70b8d80-c6a7-4498-ab2d-d1a7d3957ceb\") " pod="openshift-multus/multus-6hk44"
Jul 23 17:05:06.688940 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.688205    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c929500-877b-43d7-9710-0ea27db45055-tmp-dir\") pod \"node-resolver-stw6g\" (UID: \"5c929500-877b-43d7-9710-0ea27db45055\") " pod="openshift-dns/node-resolver-stw6g"
Jul 23 17:05:06.688940 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.688216    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/65acd752-a782-40a8-82b9-3891d7349f33-system-cni-dir\") pod \"multus-additional-cni-plugins-b797v\" (UID: \"65acd752-a782-40a8-82b9-3891d7349f33\") " pod="openshift-multus/multus-additional-cni-plugins-b797v"
Jul 23 17:05:06.688940 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.688241    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/62550ab4-e316-461d-a09e-679202149bb2-etc-sysctl-d\") pod \"tuned-vj6t5\" (UID: \"62550ab4-e316-461d-a09e-679202149bb2\") " pod="openshift-cluster-node-tuning-operator/tuned-vj6t5"
Jul 23 17:05:06.688940 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.688265    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kgbvc\" (UniqueName: \"kubernetes.io/projected/564dda72-0cfc-4e30-b04d-eddd1a9b7b30-kube-api-access-kgbvc\") pod \"node-ca-znmk5\" (UID: \"564dda72-0cfc-4e30-b04d-eddd1a9b7b30\") " pod="openshift-image-registry/node-ca-znmk5"
Jul 23 17:05:06.688940 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.688286    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/65acd752-a782-40a8-82b9-3891d7349f33-system-cni-dir\") pod \"multus-additional-cni-plugins-b797v\" (UID: \"65acd752-a782-40a8-82b9-3891d7349f33\") " pod="openshift-multus/multus-additional-cni-plugins-b797v"
Jul 23 17:05:06.688940 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.688290    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a70b8d80-c6a7-4498-ab2d-d1a7d3957ceb-cni-binary-copy\") pod \"multus-6hk44\" (UID: \"a70b8d80-c6a7-4498-ab2d-d1a7d3957ceb\") " pod="openshift-multus/multus-6hk44"
Jul 23 17:05:06.688940 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.688345    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a70b8d80-c6a7-4498-ab2d-d1a7d3957ceb-host-run-multus-certs\") pod \"multus-6hk44\" (UID: \"a70b8d80-c6a7-4498-ab2d-d1a7d3957ceb\") " pod="openshift-multus/multus-6hk44"
Jul 23 17:05:06.688940 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.688377    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/71b6eaa7-70df-4f01-b0e9-6be6043acbc4-registration-dir\") pod \"aws-ebs-csi-driver-node-p4twn\" (UID: \"71b6eaa7-70df-4f01-b0e9-6be6043acbc4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p4twn"
Jul 23 17:05:06.688940 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.688404    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/71b6eaa7-70df-4f01-b0e9-6be6043acbc4-etc-selinux\") pod \"aws-ebs-csi-driver-node-p4twn\" (UID: \"71b6eaa7-70df-4f01-b0e9-6be6043acbc4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p4twn"
Jul 23 17:05:06.688940 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.688433    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c5765452-37b1-4153-9096-778d6b33857c-host-kubelet\") pod \"ovnkube-node-2s9nz\" (UID: \"c5765452-37b1-4153-9096-778d6b33857c\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s9nz"
Jul 23 17:05:06.688940 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.688459    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c5765452-37b1-4153-9096-778d6b33857c-host-run-ovn-kubernetes\") pod \"ovnkube-node-2s9nz\" (UID: \"c5765452-37b1-4153-9096-778d6b33857c\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s9nz"
Jul 23 17:05:06.688940 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.688486    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c5765452-37b1-4153-9096-778d6b33857c-ovnkube-config\") pod \"ovnkube-node-2s9nz\" (UID: \"c5765452-37b1-4153-9096-778d6b33857c\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s9nz"
Jul 23 17:05:06.688940 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.688501    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/71b6eaa7-70df-4f01-b0e9-6be6043acbc4-registration-dir\") pod \"aws-ebs-csi-driver-node-p4twn\" (UID: \"71b6eaa7-70df-4f01-b0e9-6be6043acbc4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p4twn"
Jul 23 17:05:06.688940 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.688512    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/62550ab4-e316-461d-a09e-679202149bb2-etc-systemd\") pod \"tuned-vj6t5\" (UID: \"62550ab4-e316-461d-a09e-679202149bb2\") " pod="openshift-cluster-node-tuning-operator/tuned-vj6t5"
Jul 23 17:05:06.688940 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.688537    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/62550ab4-e316-461d-a09e-679202149bb2-run\") pod \"tuned-vj6t5\" (UID: \"62550ab4-e316-461d-a09e-679202149bb2\") " pod="openshift-cluster-node-tuning-operator/tuned-vj6t5"
Jul 23 17:05:06.689763 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.688535    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/71b6eaa7-70df-4f01-b0e9-6be6043acbc4-etc-selinux\") pod \"aws-ebs-csi-driver-node-p4twn\" (UID: \"71b6eaa7-70df-4f01-b0e9-6be6043acbc4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p4twn"
Jul 23 17:05:06.689763 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.688575    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a70b8d80-c6a7-4498-ab2d-d1a7d3957ceb-etc-kubernetes\") pod \"multus-6hk44\" (UID: \"a70b8d80-c6a7-4498-ab2d-d1a7d3957ceb\") " pod="openshift-multus/multus-6hk44"
Jul 23 17:05:06.689763 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.688620    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/65acd752-a782-40a8-82b9-3891d7349f33-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-b797v\" (UID: \"65acd752-a782-40a8-82b9-3891d7349f33\") " pod="openshift-multus/multus-additional-cni-plugins-b797v"
Jul 23 17:05:06.689763 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.688654    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/9a44608a-7a55-45f6-a4dc-199578a1d620-konnectivity-ca\") pod \"konnectivity-agent-pc8xz\" (UID: \"9a44608a-7a55-45f6-a4dc-199578a1d620\") " pod="kube-system/konnectivity-agent-pc8xz"
Jul 23 17:05:06.689763 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.688686    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c5765452-37b1-4153-9096-778d6b33857c-etc-openvswitch\") pod \"ovnkube-node-2s9nz\" (UID: \"c5765452-37b1-4153-9096-778d6b33857c\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s9nz"
Jul 23 17:05:06.689763 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.688712    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c5765452-37b1-4153-9096-778d6b33857c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2s9nz\" (UID: \"c5765452-37b1-4153-9096-778d6b33857c\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s9nz"
Jul 23 17:05:06.689763 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.688728    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/564dda72-0cfc-4e30-b04d-eddd1a9b7b30-serviceca\") pod \"node-ca-znmk5\" (UID: \"564dda72-0cfc-4e30-b04d-eddd1a9b7b30\") " pod="openshift-image-registry/node-ca-znmk5"
Jul 23 17:05:06.689763 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.688743    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c5765452-37b1-4153-9096-778d6b33857c-host-slash\") pod \"ovnkube-node-2s9nz\" (UID: \"c5765452-37b1-4153-9096-778d6b33857c\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s9nz"
Jul 23 17:05:06.689763 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.688779    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/5c929500-877b-43d7-9710-0ea27db45055-hosts-file\") pod \"node-resolver-stw6g\" (UID: \"5c929500-877b-43d7-9710-0ea27db45055\") " pod="openshift-dns/node-resolver-stw6g"
Jul 23 17:05:06.689763 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.688813    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/65acd752-a782-40a8-82b9-3891d7349f33-tuning-conf-dir\") pod \"multus-additional-cni-plugins-b797v\" (UID: \"65acd752-a782-40a8-82b9-3891d7349f33\") " pod="openshift-multus/multus-additional-cni-plugins-b797v"
Jul 23 17:05:06.689763 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.688841    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/65acd752-a782-40a8-82b9-3891d7349f33-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-b797v\" (UID: \"65acd752-a782-40a8-82b9-3891d7349f33\") " pod="openshift-multus/multus-additional-cni-plugins-b797v"
Jul 23 17:05:06.689763 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.688888    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/62550ab4-e316-461d-a09e-679202149bb2-tmp\") pod \"tuned-vj6t5\" (UID: \"62550ab4-e316-461d-a09e-679202149bb2\") " pod="openshift-cluster-node-tuning-operator/tuned-vj6t5"
Jul 23 17:05:06.689763 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.688894    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/5c929500-877b-43d7-9710-0ea27db45055-hosts-file\") pod \"node-resolver-stw6g\" (UID: \"5c929500-877b-43d7-9710-0ea27db45055\") " pod="openshift-dns/node-resolver-stw6g"
Jul 23 17:05:06.689763 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.688914    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a70b8d80-c6a7-4498-ab2d-d1a7d3957ceb-multus-cni-dir\") pod \"multus-6hk44\" (UID: \"a70b8d80-c6a7-4498-ab2d-d1a7d3957ceb\") " pod="openshift-multus/multus-6hk44"
Jul 23 17:05:06.689763 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.688986    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/65acd752-a782-40a8-82b9-3891d7349f33-tuning-conf-dir\") pod \"multus-additional-cni-plugins-b797v\" (UID: \"65acd752-a782-40a8-82b9-3891d7349f33\") " pod="openshift-multus/multus-additional-cni-plugins-b797v"
Jul 23 17:05:06.689763 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.689044    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/664f5a33-efcf-447f-8e05-a3f6ce7e4f07-metrics-certs\") pod \"network-metrics-daemon-mjfnh\" (UID: \"664f5a33-efcf-447f-8e05-a3f6ce7e4f07\") " pod="openshift-multus/network-metrics-daemon-mjfnh"
Jul 23 17:05:06.689763 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.689077    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/65acd752-a782-40a8-82b9-3891d7349f33-cnibin\") pod \"multus-additional-cni-plugins-b797v\" (UID: \"65acd752-a782-40a8-82b9-3891d7349f33\") " pod="openshift-multus/multus-additional-cni-plugins-b797v"
Jul 23 17:05:06.690424 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.689110    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/88de4e3c-bd4f-44d7-a9aa-58c2c4d24549-iptables-alerter-script\") pod \"iptables-alerter-ft9nr\" (UID: \"88de4e3c-bd4f-44d7-a9aa-58c2c4d24549\") " pod="openshift-network-operator/iptables-alerter-ft9nr"
Jul 23 17:05:06.690424 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.689140    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qjcc\" (UniqueName: \"kubernetes.io/projected/88de4e3c-bd4f-44d7-a9aa-58c2c4d24549-kube-api-access-6qjcc\") pod \"iptables-alerter-ft9nr\" (UID: \"88de4e3c-bd4f-44d7-a9aa-58c2c4d24549\") " pod="openshift-network-operator/iptables-alerter-ft9nr"
Jul 23 17:05:06.690424 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.689164    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/65acd752-a782-40a8-82b9-3891d7349f33-cnibin\") pod \"multus-additional-cni-plugins-b797v\" (UID: \"65acd752-a782-40a8-82b9-3891d7349f33\") " pod="openshift-multus/multus-additional-cni-plugins-b797v"
Jul 23 17:05:06.690424 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.689195    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c5765452-37b1-4153-9096-778d6b33857c-run-systemd\") pod \"ovnkube-node-2s9nz\" (UID: \"c5765452-37b1-4153-9096-778d6b33857c\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s9nz"
Jul 23 17:05:06.690424 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.689230    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c5765452-37b1-4153-9096-778d6b33857c-host-cni-bin\") pod \"ovnkube-node-2s9nz\" (UID: \"c5765452-37b1-4153-9096-778d6b33857c\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s9nz"
Jul 23 17:05:06.690424 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.689281    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lk7rx\" (UniqueName: \"kubernetes.io/projected/a70b8d80-c6a7-4498-ab2d-d1a7d3957ceb-kube-api-access-lk7rx\") pod \"multus-6hk44\" (UID: \"a70b8d80-c6a7-4498-ab2d-d1a7d3957ceb\") " pod="openshift-multus/multus-6hk44"
Jul 23 17:05:06.690424 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.689317    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/71b6eaa7-70df-4f01-b0e9-6be6043acbc4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-p4twn\" (UID: \"71b6eaa7-70df-4f01-b0e9-6be6043acbc4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p4twn"
Jul 23 17:05:06.690424 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.689341    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/65acd752-a782-40a8-82b9-3891d7349f33-cni-binary-copy\") pod \"multus-additional-cni-plugins-b797v\" (UID: \"65acd752-a782-40a8-82b9-3891d7349f33\") " pod="openshift-multus/multus-additional-cni-plugins-b797v"
Jul 23 17:05:06.690424 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.689368    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4dcq\" (UniqueName: \"kubernetes.io/projected/65acd752-a782-40a8-82b9-3891d7349f33-kube-api-access-w4dcq\") pod \"multus-additional-cni-plugins-b797v\" (UID: \"65acd752-a782-40a8-82b9-3891d7349f33\") " pod="openshift-multus/multus-additional-cni-plugins-b797v"
Jul 23 17:05:06.690424 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.689394    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/564dda72-0cfc-4e30-b04d-eddd1a9b7b30-host\") pod \"node-ca-znmk5\" (UID: \"564dda72-0cfc-4e30-b04d-eddd1a9b7b30\") " pod="openshift-image-registry/node-ca-znmk5"
Jul 23 17:05:06.690424 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.689407    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/71b6eaa7-70df-4f01-b0e9-6be6043acbc4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-p4twn\" (UID: \"71b6eaa7-70df-4f01-b0e9-6be6043acbc4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p4twn"
Jul 23 17:05:06.690424 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.689418    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a70b8d80-c6a7-4498-ab2d-d1a7d3957ceb-os-release\") pod \"multus-6hk44\" (UID: \"a70b8d80-c6a7-4498-ab2d-d1a7d3957ceb\") " pod="openshift-multus/multus-6hk44"
Jul 23 17:05:06.690424 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.689443    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ktd2\" (UniqueName: \"kubernetes.io/projected/664f5a33-efcf-447f-8e05-a3f6ce7e4f07-kube-api-access-6ktd2\") pod \"network-metrics-daemon-mjfnh\" (UID: \"664f5a33-efcf-447f-8e05-a3f6ce7e4f07\") " pod="openshift-multus/network-metrics-daemon-mjfnh"
Jul 23 17:05:06.690424 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.689467    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/71b6eaa7-70df-4f01-b0e9-6be6043acbc4-sys-fs\") pod \"aws-ebs-csi-driver-node-p4twn\" (UID: \"71b6eaa7-70df-4f01-b0e9-6be6043acbc4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p4twn"
Jul 23 17:05:06.690424 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.689494    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/9a44608a-7a55-45f6-a4dc-199578a1d620-agent-certs\") pod \"konnectivity-agent-pc8xz\" (UID: \"9a44608a-7a55-45f6-a4dc-199578a1d620\") " pod="kube-system/konnectivity-agent-pc8xz"
Jul 23 17:05:06.690424 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.689518    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/62550ab4-e316-461d-a09e-679202149bb2-lib-modules\") pod \"tuned-vj6t5\" (UID: \"62550ab4-e316-461d-a09e-679202149bb2\") " pod="openshift-cluster-node-tuning-operator/tuned-vj6t5"
Jul 23 17:05:06.691003 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.689543    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a70b8d80-c6a7-4498-ab2d-d1a7d3957ceb-host-var-lib-kubelet\") pod \"multus-6hk44\" (UID: \"a70b8d80-c6a7-4498-ab2d-d1a7d3957ceb\") " pod="openshift-multus/multus-6hk44"
Jul 23 17:05:06.691003 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.689694    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/71b6eaa7-70df-4f01-b0e9-6be6043acbc4-sys-fs\") pod \"aws-ebs-csi-driver-node-p4twn\" (UID: \"71b6eaa7-70df-4f01-b0e9-6be6043acbc4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p4twn"
Jul 23 17:05:06.691003 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.689838    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/65acd752-a782-40a8-82b9-3891d7349f33-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-b797v\" (UID: \"65acd752-a782-40a8-82b9-3891d7349f33\") " pod="openshift-multus/multus-additional-cni-plugins-b797v"
Jul 23 17:05:06.691003 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.689884    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/65acd752-a782-40a8-82b9-3891d7349f33-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-b797v\" (UID: \"65acd752-a782-40a8-82b9-3891d7349f33\") " pod="openshift-multus/multus-additional-cni-plugins-b797v"
Jul 23 17:05:06.691003 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.690053    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/65acd752-a782-40a8-82b9-3891d7349f33-cni-binary-copy\") pod \"multus-additional-cni-plugins-b797v\" (UID: \"65acd752-a782-40a8-82b9-3891d7349f33\") " pod="openshift-multus/multus-additional-cni-plugins-b797v"
Jul 23 17:05:06.711113 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.711084    2577 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 23 17:05:06.715111 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.715066    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-twxbm\" (UniqueName: \"kubernetes.io/projected/5c929500-877b-43d7-9710-0ea27db45055-kube-api-access-twxbm\") pod \"node-resolver-stw6g\" (UID: \"5c929500-877b-43d7-9710-0ea27db45055\") " pod="openshift-dns/node-resolver-stw6g"
Jul 23 17:05:06.715111 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.715096    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4dcq\" (UniqueName: \"kubernetes.io/projected/65acd752-a782-40a8-82b9-3891d7349f33-kube-api-access-w4dcq\") pod \"multus-additional-cni-plugins-b797v\" (UID: \"65acd752-a782-40a8-82b9-3891d7349f33\") " pod="openshift-multus/multus-additional-cni-plugins-b797v"
Jul 23 17:05:06.715289 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.715095    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zs94q\" (UniqueName: \"kubernetes.io/projected/71b6eaa7-70df-4f01-b0e9-6be6043acbc4-kube-api-access-zs94q\") pod \"aws-ebs-csi-driver-node-p4twn\" (UID: \"71b6eaa7-70df-4f01-b0e9-6be6043acbc4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p4twn"
Jul 23 17:05:06.721711 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:06.721690    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28346eafc8fa850eaa3541eee81eaa75.slice/crio-dbaf65e5ed3798ca5ecf06dee7489a0618e1f9f09b188d05601e9045c029c83c WatchSource:0}: Error finding container dbaf65e5ed3798ca5ecf06dee7489a0618e1f9f09b188d05601e9045c029c83c: Status 404 returned error can't find the container with id dbaf65e5ed3798ca5ecf06dee7489a0618e1f9f09b188d05601e9045c029c83c
Jul 23 17:05:06.765343 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.765290    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-216.ec2.internal" event={"ID":"28346eafc8fa850eaa3541eee81eaa75","Type":"ContainerStarted","Data":"dbaf65e5ed3798ca5ecf06dee7489a0618e1f9f09b188d05601e9045c029c83c"}
Jul 23 17:05:06.766676 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.766640    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-216.ec2.internal" event={"ID":"7bd1d5c1fbf257c4a5dcb5c0b12f7f75","Type":"ContainerStarted","Data":"38d0bb6f5de6697f84b4329230e2823bcaeb19960c94283042782fdee2b520ff"}
Jul 23 17:05:06.783901 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.783872    2577 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 23 17:05:06.790252 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.790219    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c5765452-37b1-4153-9096-778d6b33857c-host-kubelet\") pod \"ovnkube-node-2s9nz\" (UID: \"c5765452-37b1-4153-9096-778d6b33857c\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s9nz"
Jul 23 17:05:06.790385 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.790283    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c5765452-37b1-4153-9096-778d6b33857c-host-run-ovn-kubernetes\") pod \"ovnkube-node-2s9nz\" (UID: \"c5765452-37b1-4153-9096-778d6b33857c\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s9nz"
Jul 23 17:05:06.790385 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.790312    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c5765452-37b1-4153-9096-778d6b33857c-ovnkube-config\") pod \"ovnkube-node-2s9nz\" (UID: \"c5765452-37b1-4153-9096-778d6b33857c\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s9nz"
Jul 23 17:05:06.790385 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.790338    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/62550ab4-e316-461d-a09e-679202149bb2-etc-systemd\") pod \"tuned-vj6t5\" (UID: \"62550ab4-e316-461d-a09e-679202149bb2\") " pod="openshift-cluster-node-tuning-operator/tuned-vj6t5"
Jul 23 17:05:06.790385 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.790348    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c5765452-37b1-4153-9096-778d6b33857c-host-kubelet\") pod \"ovnkube-node-2s9nz\" (UID: \"c5765452-37b1-4153-9096-778d6b33857c\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s9nz"
Jul 23 17:05:06.790385 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.790362    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/62550ab4-e316-461d-a09e-679202149bb2-run\") pod \"tuned-vj6t5\" (UID: \"62550ab4-e316-461d-a09e-679202149bb2\") " pod="openshift-cluster-node-tuning-operator/tuned-vj6t5"
Jul 23 17:05:06.790692 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.790380    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c5765452-37b1-4153-9096-778d6b33857c-host-run-ovn-kubernetes\") pod \"ovnkube-node-2s9nz\" (UID: \"c5765452-37b1-4153-9096-778d6b33857c\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s9nz"
Jul 23 17:05:06.790692 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.790386    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a70b8d80-c6a7-4498-ab2d-d1a7d3957ceb-etc-kubernetes\") pod \"multus-6hk44\" (UID: \"a70b8d80-c6a7-4498-ab2d-d1a7d3957ceb\") " pod="openshift-multus/multus-6hk44"
Jul 23 17:05:06.790692 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.790418    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a70b8d80-c6a7-4498-ab2d-d1a7d3957ceb-etc-kubernetes\") pod \"multus-6hk44\" (UID: \"a70b8d80-c6a7-4498-ab2d-d1a7d3957ceb\") " pod="openshift-multus/multus-6hk44"
Jul 23 17:05:06.790692 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.790454    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/9a44608a-7a55-45f6-a4dc-199578a1d620-konnectivity-ca\") pod \"konnectivity-agent-pc8xz\" (UID: \"9a44608a-7a55-45f6-a4dc-199578a1d620\") " pod="kube-system/konnectivity-agent-pc8xz"
Jul 23 17:05:06.790692 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.790472    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/62550ab4-e316-461d-a09e-679202149bb2-etc-systemd\") pod \"tuned-vj6t5\" (UID: \"62550ab4-e316-461d-a09e-679202149bb2\") " pod="openshift-cluster-node-tuning-operator/tuned-vj6t5"
Jul 23 17:05:06.790692 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.790483    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c5765452-37b1-4153-9096-778d6b33857c-etc-openvswitch\") pod \"ovnkube-node-2s9nz\" (UID: \"c5765452-37b1-4153-9096-778d6b33857c\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s9nz"
Jul 23 17:05:06.790692 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.790535    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c5765452-37b1-4153-9096-778d6b33857c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2s9nz\" (UID: \"c5765452-37b1-4153-9096-778d6b33857c\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s9nz"
Jul 23 17:05:06.790692 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.790538    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/62550ab4-e316-461d-a09e-679202149bb2-run\") pod \"tuned-vj6t5\" (UID: \"62550ab4-e316-461d-a09e-679202149bb2\") " pod="openshift-cluster-node-tuning-operator/tuned-vj6t5"
Jul 23 17:05:06.790692 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.790564    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/564dda72-0cfc-4e30-b04d-eddd1a9b7b30-serviceca\") pod \"node-ca-znmk5\" (UID: \"564dda72-0cfc-4e30-b04d-eddd1a9b7b30\") " pod="openshift-image-registry/node-ca-znmk5"
Jul 23 17:05:06.790692 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.790601    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c5765452-37b1-4153-9096-778d6b33857c-etc-openvswitch\") pod \"ovnkube-node-2s9nz\" (UID: \"c5765452-37b1-4153-9096-778d6b33857c\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s9nz"
Jul 23 17:05:06.790692 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.790601    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c5765452-37b1-4153-9096-778d6b33857c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2s9nz\" (UID: \"c5765452-37b1-4153-9096-778d6b33857c\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s9nz"
Jul 23 17:05:06.790692 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.790588    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c5765452-37b1-4153-9096-778d6b33857c-host-slash\") pod \"ovnkube-node-2s9nz\" (UID: \"c5765452-37b1-4153-9096-778d6b33857c\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s9nz"
Jul 23 17:05:06.790692 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.790659    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c5765452-37b1-4153-9096-778d6b33857c-host-slash\") pod \"ovnkube-node-2s9nz\" (UID: \"c5765452-37b1-4153-9096-778d6b33857c\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s9nz"
Jul 23 17:05:06.790692 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.790663    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/62550ab4-e316-461d-a09e-679202149bb2-tmp\") pod \"tuned-vj6t5\" (UID: \"62550ab4-e316-461d-a09e-679202149bb2\") " pod="openshift-cluster-node-tuning-operator/tuned-vj6t5"
Jul 23 17:05:06.790692 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.790690    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a70b8d80-c6a7-4498-ab2d-d1a7d3957ceb-multus-cni-dir\") pod \"multus-6hk44\" (UID: \"a70b8d80-c6a7-4498-ab2d-d1a7d3957ceb\") " pod="openshift-multus/multus-6hk44"
Jul 23 17:05:06.791463 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.790721    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/664f5a33-efcf-447f-8e05-a3f6ce7e4f07-metrics-certs\") pod \"network-metrics-daemon-mjfnh\" (UID: \"664f5a33-efcf-447f-8e05-a3f6ce7e4f07\") " pod="openshift-multus/network-metrics-daemon-mjfnh"
Jul 23 17:05:06.791463 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.790749    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/88de4e3c-bd4f-44d7-a9aa-58c2c4d24549-iptables-alerter-script\") pod \"iptables-alerter-ft9nr\" (UID: \"88de4e3c-bd4f-44d7-a9aa-58c2c4d24549\") " pod="openshift-network-operator/iptables-alerter-ft9nr"
Jul 23 17:05:06.791463 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.790776    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qjcc\" (UniqueName: \"kubernetes.io/projected/88de4e3c-bd4f-44d7-a9aa-58c2c4d24549-kube-api-access-6qjcc\") pod \"iptables-alerter-ft9nr\" (UID: \"88de4e3c-bd4f-44d7-a9aa-58c2c4d24549\") " pod="openshift-network-operator/iptables-alerter-ft9nr"
Jul 23 17:05:06.791463 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.790800    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c5765452-37b1-4153-9096-778d6b33857c-run-systemd\") pod \"ovnkube-node-2s9nz\" (UID: \"c5765452-37b1-4153-9096-778d6b33857c\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s9nz"
Jul 23 17:05:06.791463 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.790824    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c5765452-37b1-4153-9096-778d6b33857c-host-cni-bin\") pod \"ovnkube-node-2s9nz\" (UID: \"c5765452-37b1-4153-9096-778d6b33857c\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s9nz"
Jul 23 17:05:06.791463 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.790850    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lk7rx\" (UniqueName: \"kubernetes.io/projected/a70b8d80-c6a7-4498-ab2d-d1a7d3957ceb-kube-api-access-lk7rx\") pod \"multus-6hk44\" (UID: \"a70b8d80-c6a7-4498-ab2d-d1a7d3957ceb\") " pod="openshift-multus/multus-6hk44"
Jul 23 17:05:06.791463 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.790875    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/564dda72-0cfc-4e30-b04d-eddd1a9b7b30-host\") pod \"node-ca-znmk5\" (UID: \"564dda72-0cfc-4e30-b04d-eddd1a9b7b30\") " pod="openshift-image-registry/node-ca-znmk5"
Jul 23 17:05:06.791463 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.790885    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a70b8d80-c6a7-4498-ab2d-d1a7d3957ceb-multus-cni-dir\") pod \"multus-6hk44\" (UID: \"a70b8d80-c6a7-4498-ab2d-d1a7d3957ceb\") " pod="openshift-multus/multus-6hk44"
Jul 23 17:05:06.791463 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.790896    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a70b8d80-c6a7-4498-ab2d-d1a7d3957ceb-os-release\") pod \"multus-6hk44\" (UID: \"a70b8d80-c6a7-4498-ab2d-d1a7d3957ceb\") " pod="openshift-multus/multus-6hk44"
Jul 23 17:05:06.791463 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.790922    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ktd2\" (UniqueName: \"kubernetes.io/projected/664f5a33-efcf-447f-8e05-a3f6ce7e4f07-kube-api-access-6ktd2\") pod \"network-metrics-daemon-mjfnh\" (UID: \"664f5a33-efcf-447f-8e05-a3f6ce7e4f07\") " pod="openshift-multus/network-metrics-daemon-mjfnh"
Jul 23 17:05:06.791463 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.790938    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c5765452-37b1-4153-9096-778d6b33857c-run-systemd\") pod \"ovnkube-node-2s9nz\" (UID: \"c5765452-37b1-4153-9096-778d6b33857c\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s9nz"
Jul 23 17:05:06.791463 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.790949    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/9a44608a-7a55-45f6-a4dc-199578a1d620-agent-certs\") pod \"konnectivity-agent-pc8xz\" (UID: \"9a44608a-7a55-45f6-a4dc-199578a1d620\") " pod="kube-system/konnectivity-agent-pc8xz"
Jul 23 17:05:06.791463 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.790975    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/62550ab4-e316-461d-a09e-679202149bb2-lib-modules\") pod \"tuned-vj6t5\" (UID: \"62550ab4-e316-461d-a09e-679202149bb2\") " pod="openshift-cluster-node-tuning-operator/tuned-vj6t5"
Jul 23 17:05:06.791463 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.790992    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c5765452-37b1-4153-9096-778d6b33857c-ovnkube-config\") pod \"ovnkube-node-2s9nz\" (UID: \"c5765452-37b1-4153-9096-778d6b33857c\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s9nz"
Jul 23 17:05:06.791463 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.791001    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a70b8d80-c6a7-4498-ab2d-d1a7d3957ceb-host-var-lib-kubelet\") pod \"multus-6hk44\" (UID: \"a70b8d80-c6a7-4498-ab2d-d1a7d3957ceb\") " pod="openshift-multus/multus-6hk44"
Jul 23 17:05:06.791463 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.791029    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hpn22\" (UniqueName: \"kubernetes.io/projected/2090b02e-14f1-4224-8691-8fa4ecf7d53b-kube-api-access-hpn22\") pod \"network-check-target-5c6b5\" (UID: \"2090b02e-14f1-4224-8691-8fa4ecf7d53b\") " pod="openshift-network-diagnostics/network-check-target-5c6b5"
Jul 23 17:05:06.791463 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:05:06.791053    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 23 17:05:06.791463 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.791055    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/62550ab4-e316-461d-a09e-679202149bb2-etc-sysconfig\") pod \"tuned-vj6t5\" (UID: \"62550ab4-e316-461d-a09e-679202149bb2\") " pod="openshift-cluster-node-tuning-operator/tuned-vj6t5"
Jul 23 17:05:06.792419 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.791093    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c5765452-37b1-4153-9096-778d6b33857c-host-cni-bin\") pod \"ovnkube-node-2s9nz\" (UID: \"c5765452-37b1-4153-9096-778d6b33857c\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s9nz"
Jul 23 17:05:06.792419 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.791112    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/9a44608a-7a55-45f6-a4dc-199578a1d620-konnectivity-ca\") pod \"konnectivity-agent-pc8xz\" (UID: \"9a44608a-7a55-45f6-a4dc-199578a1d620\") " pod="kube-system/konnectivity-agent-pc8xz"
Jul 23 17:05:06.792419 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.791134    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/564dda72-0cfc-4e30-b04d-eddd1a9b7b30-host\") pod \"node-ca-znmk5\" (UID: \"564dda72-0cfc-4e30-b04d-eddd1a9b7b30\") " pod="openshift-image-registry/node-ca-znmk5"
Jul 23 17:05:06.792419 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.791134    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/564dda72-0cfc-4e30-b04d-eddd1a9b7b30-serviceca\") pod \"node-ca-znmk5\" (UID: \"564dda72-0cfc-4e30-b04d-eddd1a9b7b30\") " pod="openshift-image-registry/node-ca-znmk5"
Jul 23 17:05:06.792419 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:05:06.791122    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/664f5a33-efcf-447f-8e05-a3f6ce7e4f07-metrics-certs podName:664f5a33-efcf-447f-8e05-a3f6ce7e4f07 nodeName:}" failed. No retries permitted until 2026-07-23 17:05:07.29110071 +0000 UTC m=+3.061386227 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/664f5a33-efcf-447f-8e05-a3f6ce7e4f07-metrics-certs") pod "network-metrics-daemon-mjfnh" (UID: "664f5a33-efcf-447f-8e05-a3f6ce7e4f07") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 23 17:05:06.792419 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.791241    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/62550ab4-e316-461d-a09e-679202149bb2-lib-modules\") pod \"tuned-vj6t5\" (UID: \"62550ab4-e316-461d-a09e-679202149bb2\") " pod="openshift-cluster-node-tuning-operator/tuned-vj6t5"
Jul 23 17:05:06.792419 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.791056    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a70b8d80-c6a7-4498-ab2d-d1a7d3957ceb-os-release\") pod \"multus-6hk44\" (UID: \"a70b8d80-c6a7-4498-ab2d-d1a7d3957ceb\") " pod="openshift-multus/multus-6hk44"
Jul 23 17:05:06.792419 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.791333    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/62550ab4-e316-461d-a09e-679202149bb2-sys\") pod \"tuned-vj6t5\" (UID: \"62550ab4-e316-461d-a09e-679202149bb2\") " pod="openshift-cluster-node-tuning-operator/tuned-vj6t5"
Jul 23 17:05:06.792419 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.791368    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a70b8d80-c6a7-4498-ab2d-d1a7d3957ceb-host-var-lib-kubelet\") pod \"multus-6hk44\" (UID: \"a70b8d80-c6a7-4498-ab2d-d1a7d3957ceb\") " pod="openshift-multus/multus-6hk44"
Jul 23 17:05:06.792419 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.791391    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a70b8d80-c6a7-4498-ab2d-d1a7d3957ceb-host-var-lib-cni-bin\") pod \"multus-6hk44\" (UID: \"a70b8d80-c6a7-4498-ab2d-d1a7d3957ceb\") " pod="openshift-multus/multus-6hk44"
Jul 23 17:05:06.792419 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.791415    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/62550ab4-e316-461d-a09e-679202149bb2-etc-sysconfig\") pod \"tuned-vj6t5\" (UID: \"62550ab4-e316-461d-a09e-679202149bb2\") " pod="openshift-cluster-node-tuning-operator/tuned-vj6t5"
Jul 23 17:05:06.792419 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.791443    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/62550ab4-e316-461d-a09e-679202149bb2-sys\") pod \"tuned-vj6t5\" (UID: \"62550ab4-e316-461d-a09e-679202149bb2\") " pod="openshift-cluster-node-tuning-operator/tuned-vj6t5"
Jul 23 17:05:06.792419 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.791446    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c5765452-37b1-4153-9096-778d6b33857c-run-openvswitch\") pod \"ovnkube-node-2s9nz\" (UID: \"c5765452-37b1-4153-9096-778d6b33857c\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s9nz"
Jul 23 17:05:06.792419 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.791461    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a70b8d80-c6a7-4498-ab2d-d1a7d3957ceb-host-var-lib-cni-bin\") pod \"multus-6hk44\" (UID: \"a70b8d80-c6a7-4498-ab2d-d1a7d3957ceb\") " pod="openshift-multus/multus-6hk44"
Jul 23 17:05:06.792419 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.791485    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c5765452-37b1-4153-9096-778d6b33857c-log-socket\") pod \"ovnkube-node-2s9nz\" (UID: \"c5765452-37b1-4153-9096-778d6b33857c\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s9nz"
Jul 23 17:05:06.792419 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.791486    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c5765452-37b1-4153-9096-778d6b33857c-run-openvswitch\") pod \"ovnkube-node-2s9nz\" (UID: \"c5765452-37b1-4153-9096-778d6b33857c\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s9nz"
Jul 23 17:05:06.792419 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.791525    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c5765452-37b1-4153-9096-778d6b33857c-log-socket\") pod \"ovnkube-node-2s9nz\" (UID: \"c5765452-37b1-4153-9096-778d6b33857c\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s9nz"
Jul 23 17:05:06.793282 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.791524    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvl55\" (UniqueName: \"kubernetes.io/projected/c5765452-37b1-4153-9096-778d6b33857c-kube-api-access-rvl55\") pod \"ovnkube-node-2s9nz\" (UID: \"c5765452-37b1-4153-9096-778d6b33857c\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s9nz"
Jul 23 17:05:06.793282 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.791557    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a70b8d80-c6a7-4498-ab2d-d1a7d3957ceb-system-cni-dir\") pod \"multus-6hk44\" (UID: \"a70b8d80-c6a7-4498-ab2d-d1a7d3957ceb\") " pod="openshift-multus/multus-6hk44"
Jul 23 17:05:06.793282 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.791582    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a70b8d80-c6a7-4498-ab2d-d1a7d3957ceb-hostroot\") pod \"multus-6hk44\" (UID: \"a70b8d80-c6a7-4498-ab2d-d1a7d3957ceb\") " pod="openshift-multus/multus-6hk44"
Jul 23 17:05:06.793282 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.791600    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a70b8d80-c6a7-4498-ab2d-d1a7d3957ceb-system-cni-dir\") pod \"multus-6hk44\" (UID: \"a70b8d80-c6a7-4498-ab2d-d1a7d3957ceb\") " pod="openshift-multus/multus-6hk44"
Jul 23 17:05:06.793282 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.791608    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/62550ab4-e316-461d-a09e-679202149bb2-etc-tuned\") pod \"tuned-vj6t5\" (UID: \"62550ab4-e316-461d-a09e-679202149bb2\") " pod="openshift-cluster-node-tuning-operator/tuned-vj6t5"
Jul 23 17:05:06.793282 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.791632    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c5765452-37b1-4153-9096-778d6b33857c-host-cni-netd\") pod \"ovnkube-node-2s9nz\" (UID: \"c5765452-37b1-4153-9096-778d6b33857c\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s9nz"
Jul 23 17:05:06.793282 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.791653    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a70b8d80-c6a7-4498-ab2d-d1a7d3957ceb-hostroot\") pod \"multus-6hk44\" (UID: \"a70b8d80-c6a7-4498-ab2d-d1a7d3957ceb\") " pod="openshift-multus/multus-6hk44"
Jul 23 17:05:06.793282 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.791658    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c5765452-37b1-4153-9096-778d6b33857c-ovn-node-metrics-cert\") pod \"ovnkube-node-2s9nz\" (UID: \"c5765452-37b1-4153-9096-778d6b33857c\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s9nz"
Jul 23 17:05:06.793282 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.791683    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/62550ab4-e316-461d-a09e-679202149bb2-etc-kubernetes\") pod \"tuned-vj6t5\" (UID: \"62550ab4-e316-461d-a09e-679202149bb2\") " pod="openshift-cluster-node-tuning-operator/tuned-vj6t5"
Jul 23 17:05:06.793282 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.791707    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/62550ab4-e316-461d-a09e-679202149bb2-etc-sysctl-conf\") pod \"tuned-vj6t5\" (UID: \"62550ab4-e316-461d-a09e-679202149bb2\") " pod="openshift-cluster-node-tuning-operator/tuned-vj6t5"
Jul 23 17:05:06.793282 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.791713    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c5765452-37b1-4153-9096-778d6b33857c-host-cni-netd\") pod \"ovnkube-node-2s9nz\" (UID: \"c5765452-37b1-4153-9096-778d6b33857c\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s9nz"
Jul 23 17:05:06.793282 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.791731    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/62550ab4-e316-461d-a09e-679202149bb2-var-lib-kubelet\") pod \"tuned-vj6t5\" (UID: \"62550ab4-e316-461d-a09e-679202149bb2\") " pod="openshift-cluster-node-tuning-operator/tuned-vj6t5"
Jul 23 17:05:06.793282 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.791757    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a70b8d80-c6a7-4498-ab2d-d1a7d3957ceb-host-var-lib-cni-multus\") pod \"multus-6hk44\" (UID: \"a70b8d80-c6a7-4498-ab2d-d1a7d3957ceb\") " pod="openshift-multus/multus-6hk44"
Jul 23 17:05:06.793282 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.791760    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/62550ab4-e316-461d-a09e-679202149bb2-etc-kubernetes\") pod \"tuned-vj6t5\" (UID: \"62550ab4-e316-461d-a09e-679202149bb2\") " pod="openshift-cluster-node-tuning-operator/tuned-vj6t5"
Jul 23 17:05:06.793282 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.791778    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/88de4e3c-bd4f-44d7-a9aa-58c2c4d24549-iptables-alerter-script\") pod \"iptables-alerter-ft9nr\" (UID: \"88de4e3c-bd4f-44d7-a9aa-58c2c4d24549\") " pod="openshift-network-operator/iptables-alerter-ft9nr"
Jul 23 17:05:06.793282 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.791783    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c5765452-37b1-4153-9096-778d6b33857c-systemd-units\") pod \"ovnkube-node-2s9nz\" (UID: \"c5765452-37b1-4153-9096-778d6b33857c\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s9nz"
Jul 23 17:05:06.793282 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.791815    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c5765452-37b1-4153-9096-778d6b33857c-systemd-units\") pod \"ovnkube-node-2s9nz\" (UID: \"c5765452-37b1-4153-9096-778d6b33857c\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s9nz"
Jul 23 17:05:06.793282 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.791840    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/88de4e3c-bd4f-44d7-a9aa-58c2c4d24549-host-slash\") pod \"iptables-alerter-ft9nr\" (UID: \"88de4e3c-bd4f-44d7-a9aa-58c2c4d24549\") " pod="openshift-network-operator/iptables-alerter-ft9nr"
Jul 23 17:05:06.794132 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.791850    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a70b8d80-c6a7-4498-ab2d-d1a7d3957ceb-host-var-lib-cni-multus\") pod \"multus-6hk44\" (UID: \"a70b8d80-c6a7-4498-ab2d-d1a7d3957ceb\") " pod="openshift-multus/multus-6hk44"
Jul 23 17:05:06.794132 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.791868    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c5765452-37b1-4153-9096-778d6b33857c-var-lib-openvswitch\") pod \"ovnkube-node-2s9nz\" (UID: \"c5765452-37b1-4153-9096-778d6b33857c\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s9nz"
Jul 23 17:05:06.794132 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.791883    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/88de4e3c-bd4f-44d7-a9aa-58c2c4d24549-host-slash\") pod \"iptables-alerter-ft9nr\" (UID: \"88de4e3c-bd4f-44d7-a9aa-58c2c4d24549\") " pod="openshift-network-operator/iptables-alerter-ft9nr"
Jul 23 17:05:06.794132 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.791808    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/62550ab4-e316-461d-a09e-679202149bb2-var-lib-kubelet\") pod \"tuned-vj6t5\" (UID: \"62550ab4-e316-461d-a09e-679202149bb2\") " pod="openshift-cluster-node-tuning-operator/tuned-vj6t5"
Jul 23 17:05:06.794132 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.791892    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c5765452-37b1-4153-9096-778d6b33857c-env-overrides\") pod \"ovnkube-node-2s9nz\" (UID: \"c5765452-37b1-4153-9096-778d6b33857c\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s9nz"
Jul 23 17:05:06.794132 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.791914    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/62550ab4-e316-461d-a09e-679202149bb2-etc-sysctl-conf\") pod \"tuned-vj6t5\" (UID: \"62550ab4-e316-461d-a09e-679202149bb2\") " pod="openshift-cluster-node-tuning-operator/tuned-vj6t5"
Jul 23 17:05:06.794132 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.791929    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a70b8d80-c6a7-4498-ab2d-d1a7d3957ceb-multus-socket-dir-parent\") pod \"multus-6hk44\" (UID: \"a70b8d80-c6a7-4498-ab2d-d1a7d3957ceb\") " pod="openshift-multus/multus-6hk44"
Jul 23 17:05:06.794132 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.791960    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c5765452-37b1-4153-9096-778d6b33857c-host-run-netns\") pod \"ovnkube-node-2s9nz\" (UID: \"c5765452-37b1-4153-9096-778d6b33857c\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s9nz"
Jul 23 17:05:06.794132 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.791979    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c5765452-37b1-4153-9096-778d6b33857c-var-lib-openvswitch\") pod \"ovnkube-node-2s9nz\" (UID: \"c5765452-37b1-4153-9096-778d6b33857c\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s9nz"
Jul 23 17:05:06.794132 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.791991    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c5765452-37b1-4153-9096-778d6b33857c-run-ovn\") pod \"ovnkube-node-2s9nz\" (UID: \"c5765452-37b1-4153-9096-778d6b33857c\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s9nz"
Jul 23 17:05:06.794132 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.792045    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c5765452-37b1-4153-9096-778d6b33857c-node-log\") pod \"ovnkube-node-2s9nz\" (UID: \"c5765452-37b1-4153-9096-778d6b33857c\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s9nz"
Jul 23 17:05:06.794132 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.792074    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/aeb10e6f-ad01-4a83-8090-049d9e919696-kubelet-config\") pod \"global-pull-secret-syncer-5g97l\" (UID: \"aeb10e6f-ad01-4a83-8090-049d9e919696\") " pod="kube-system/global-pull-secret-syncer-5g97l"
Jul 23 17:05:06.794132 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.792101    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/aeb10e6f-ad01-4a83-8090-049d9e919696-dbus\") pod \"global-pull-secret-syncer-5g97l\" (UID: \"aeb10e6f-ad01-4a83-8090-049d9e919696\") " pod="kube-system/global-pull-secret-syncer-5g97l"
Jul 23 17:05:06.794132 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.792119    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c5765452-37b1-4153-9096-778d6b33857c-node-log\") pod \"ovnkube-node-2s9nz\" (UID: \"c5765452-37b1-4153-9096-778d6b33857c\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s9nz"
Jul 23 17:05:06.794132 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.792130    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/aeb10e6f-ad01-4a83-8090-049d9e919696-original-pull-secret\") pod \"global-pull-secret-syncer-5g97l\" (UID: \"aeb10e6f-ad01-4a83-8090-049d9e919696\") " pod="kube-system/global-pull-secret-syncer-5g97l"
Jul 23 17:05:06.794132 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.792158    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/62550ab4-e316-461d-a09e-679202149bb2-etc-modprobe-d\") pod \"tuned-vj6t5\" (UID: \"62550ab4-e316-461d-a09e-679202149bb2\") " pod="openshift-cluster-node-tuning-operator/tuned-vj6t5"
Jul 23 17:05:06.794132 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.792161    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c5765452-37b1-4153-9096-778d6b33857c-host-run-netns\") pod \"ovnkube-node-2s9nz\" (UID: \"c5765452-37b1-4153-9096-778d6b33857c\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s9nz"
Jul 23 17:05:06.794132 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.792186    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/62550ab4-e316-461d-a09e-679202149bb2-host\") pod \"tuned-vj6t5\" (UID: \"62550ab4-e316-461d-a09e-679202149bb2\") " pod="openshift-cluster-node-tuning-operator/tuned-vj6t5"
Jul 23 17:05:06.795018 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.792213    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvc29\" (UniqueName: \"kubernetes.io/projected/62550ab4-e316-461d-a09e-679202149bb2-kube-api-access-nvc29\") pod \"tuned-vj6t5\" (UID: \"62550ab4-e316-461d-a09e-679202149bb2\") " pod="openshift-cluster-node-tuning-operator/tuned-vj6t5"
Jul 23 17:05:06.795018 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.792243    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a70b8d80-c6a7-4498-ab2d-d1a7d3957ceb-cnibin\") pod \"multus-6hk44\" (UID: \"a70b8d80-c6a7-4498-ab2d-d1a7d3957ceb\") " pod="openshift-multus/multus-6hk44"
Jul 23 17:05:06.795018 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.792245    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c5765452-37b1-4153-9096-778d6b33857c-env-overrides\") pod \"ovnkube-node-2s9nz\" (UID: \"c5765452-37b1-4153-9096-778d6b33857c\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s9nz"
Jul 23 17:05:06.795018 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.792256    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/aeb10e6f-ad01-4a83-8090-049d9e919696-kubelet-config\") pod \"global-pull-secret-syncer-5g97l\" (UID: \"aeb10e6f-ad01-4a83-8090-049d9e919696\") " pod="kube-system/global-pull-secret-syncer-5g97l"
Jul 23 17:05:06.795018 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.792267    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a70b8d80-c6a7-4498-ab2d-d1a7d3957ceb-host-run-k8s-cni-cncf-io\") pod \"multus-6hk44\" (UID: \"a70b8d80-c6a7-4498-ab2d-d1a7d3957ceb\") " pod="openshift-multus/multus-6hk44"
Jul 23 17:05:06.795018 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.792212    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a70b8d80-c6a7-4498-ab2d-d1a7d3957ceb-multus-socket-dir-parent\") pod \"multus-6hk44\" (UID: \"a70b8d80-c6a7-4498-ab2d-d1a7d3957ceb\") " pod="openshift-multus/multus-6hk44"
Jul 23 17:05:06.795018 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.792310    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a70b8d80-c6a7-4498-ab2d-d1a7d3957ceb-multus-daemon-config\") pod \"multus-6hk44\" (UID: \"a70b8d80-c6a7-4498-ab2d-d1a7d3957ceb\") " pod="openshift-multus/multus-6hk44"
Jul 23 17:05:06.795018 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.792847    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a70b8d80-c6a7-4498-ab2d-d1a7d3957ceb-multus-daemon-config\") pod \"multus-6hk44\" (UID: \"a70b8d80-c6a7-4498-ab2d-d1a7d3957ceb\") " pod="openshift-multus/multus-6hk44"
Jul 23 17:05:06.795018 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.792871    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c5765452-37b1-4153-9096-778d6b33857c-ovnkube-script-lib\") pod \"ovnkube-node-2s9nz\" (UID: \"c5765452-37b1-4153-9096-778d6b33857c\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s9nz"
Jul 23 17:05:06.795018 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.792919    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a70b8d80-c6a7-4498-ab2d-d1a7d3957ceb-host-run-netns\") pod \"multus-6hk44\" (UID: \"a70b8d80-c6a7-4498-ab2d-d1a7d3957ceb\") " pod="openshift-multus/multus-6hk44"
Jul 23 17:05:06.795018 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.792949    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a70b8d80-c6a7-4498-ab2d-d1a7d3957ceb-multus-conf-dir\") pod \"multus-6hk44\" (UID: \"a70b8d80-c6a7-4498-ab2d-d1a7d3957ceb\") " pod="openshift-multus/multus-6hk44"
Jul 23 17:05:06.795018 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.792988    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/62550ab4-e316-461d-a09e-679202149bb2-etc-sysctl-d\") pod \"tuned-vj6t5\" (UID: \"62550ab4-e316-461d-a09e-679202149bb2\") " pod="openshift-cluster-node-tuning-operator/tuned-vj6t5"
Jul 23 17:05:06.795018 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.793015    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgbvc\" (UniqueName: \"kubernetes.io/projected/564dda72-0cfc-4e30-b04d-eddd1a9b7b30-kube-api-access-kgbvc\") pod \"node-ca-znmk5\" (UID: \"564dda72-0cfc-4e30-b04d-eddd1a9b7b30\") " pod="openshift-image-registry/node-ca-znmk5"
Jul 23 17:05:06.795018 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.793041    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a70b8d80-c6a7-4498-ab2d-d1a7d3957ceb-cni-binary-copy\") pod \"multus-6hk44\" (UID: \"a70b8d80-c6a7-4498-ab2d-d1a7d3957ceb\") " pod="openshift-multus/multus-6hk44"
Jul 23 17:05:06.795018 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.793066    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a70b8d80-c6a7-4498-ab2d-d1a7d3957ceb-host-run-multus-certs\") pod \"multus-6hk44\" (UID: \"a70b8d80-c6a7-4498-ab2d-d1a7d3957ceb\") " pod="openshift-multus/multus-6hk44"
Jul 23 17:05:06.795018 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.792322    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/62550ab4-e316-461d-a09e-679202149bb2-host\") pod \"tuned-vj6t5\" (UID: \"62550ab4-e316-461d-a09e-679202149bb2\") " pod="openshift-cluster-node-tuning-operator/tuned-vj6t5"
Jul 23 17:05:06.795018 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.792438    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/aeb10e6f-ad01-4a83-8090-049d9e919696-dbus\") pod \"global-pull-secret-syncer-5g97l\" (UID: \"aeb10e6f-ad01-4a83-8090-049d9e919696\") " pod="kube-system/global-pull-secret-syncer-5g97l"
Jul 23 17:05:06.795018 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.793172    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a70b8d80-c6a7-4498-ab2d-d1a7d3957ceb-host-run-netns\") pod \"multus-6hk44\" (UID: \"a70b8d80-c6a7-4498-ab2d-d1a7d3957ceb\") " pod="openshift-multus/multus-6hk44"
Jul 23 17:05:06.795596 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:05:06.792504    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 23 17:05:06.795596 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:05:06.793263    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aeb10e6f-ad01-4a83-8090-049d9e919696-original-pull-secret podName:aeb10e6f-ad01-4a83-8090-049d9e919696 nodeName:}" failed. No retries permitted until 2026-07-23 17:05:07.293241611 +0000 UTC m=+3.063527126 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/aeb10e6f-ad01-4a83-8090-049d9e919696-original-pull-secret") pod "global-pull-secret-syncer-5g97l" (UID: "aeb10e6f-ad01-4a83-8090-049d9e919696") : object "kube-system"/"original-pull-secret" not registered
Jul 23 17:05:06.795596 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.793313    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a70b8d80-c6a7-4498-ab2d-d1a7d3957ceb-multus-conf-dir\") pod \"multus-6hk44\" (UID: \"a70b8d80-c6a7-4498-ab2d-d1a7d3957ceb\") " pod="openshift-multus/multus-6hk44"
Jul 23 17:05:06.795596 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.792501    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a70b8d80-c6a7-4498-ab2d-d1a7d3957ceb-cnibin\") pod \"multus-6hk44\" (UID: \"a70b8d80-c6a7-4498-ab2d-d1a7d3957ceb\") " pod="openshift-multus/multus-6hk44"
Jul 23 17:05:06.795596 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.793388    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c5765452-37b1-4153-9096-778d6b33857c-ovnkube-script-lib\") pod \"ovnkube-node-2s9nz\" (UID: \"c5765452-37b1-4153-9096-778d6b33857c\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s9nz"
Jul 23 17:05:06.795596 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.793450    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/62550ab4-e316-461d-a09e-679202149bb2-etc-sysctl-d\") pod \"tuned-vj6t5\" (UID: \"62550ab4-e316-461d-a09e-679202149bb2\") " pod="openshift-cluster-node-tuning-operator/tuned-vj6t5"
Jul 23 17:05:06.795596 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.792584    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/62550ab4-e316-461d-a09e-679202149bb2-etc-modprobe-d\") pod \"tuned-vj6t5\" (UID: \"62550ab4-e316-461d-a09e-679202149bb2\") " pod="openshift-cluster-node-tuning-operator/tuned-vj6t5"
Jul 23 17:05:06.795596 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.792622    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c5765452-37b1-4153-9096-778d6b33857c-run-ovn\") pod \"ovnkube-node-2s9nz\" (UID: \"c5765452-37b1-4153-9096-778d6b33857c\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s9nz"
Jul 23 17:05:06.795596 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.792385    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a70b8d80-c6a7-4498-ab2d-d1a7d3957ceb-host-run-k8s-cni-cncf-io\") pod \"multus-6hk44\" (UID: \"a70b8d80-c6a7-4498-ab2d-d1a7d3957ceb\") " pod="openshift-multus/multus-6hk44"
Jul 23 17:05:06.795596 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.793740    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a70b8d80-c6a7-4498-ab2d-d1a7d3957ceb-host-run-multus-certs\") pod \"multus-6hk44\" (UID: \"a70b8d80-c6a7-4498-ab2d-d1a7d3957ceb\") " pod="openshift-multus/multus-6hk44"
Jul 23 17:05:06.795596 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.794191    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a70b8d80-c6a7-4498-ab2d-d1a7d3957ceb-cni-binary-copy\") pod \"multus-6hk44\" (UID: \"a70b8d80-c6a7-4498-ab2d-d1a7d3957ceb\") " pod="openshift-multus/multus-6hk44"
Jul 23 17:05:06.795596 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.794227    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/9a44608a-7a55-45f6-a4dc-199578a1d620-agent-certs\") pod \"konnectivity-agent-pc8xz\" (UID: \"9a44608a-7a55-45f6-a4dc-199578a1d620\") " pod="kube-system/konnectivity-agent-pc8xz"
Jul 23 17:05:06.795596 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.794502    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/62550ab4-e316-461d-a09e-679202149bb2-tmp\") pod \"tuned-vj6t5\" (UID: \"62550ab4-e316-461d-a09e-679202149bb2\") " pod="openshift-cluster-node-tuning-operator/tuned-vj6t5"
Jul 23 17:05:06.795596 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.794741    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/62550ab4-e316-461d-a09e-679202149bb2-etc-tuned\") pod \"tuned-vj6t5\" (UID: \"62550ab4-e316-461d-a09e-679202149bb2\") " pod="openshift-cluster-node-tuning-operator/tuned-vj6t5"
Jul 23 17:05:06.795596 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.794764    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c5765452-37b1-4153-9096-778d6b33857c-ovn-node-metrics-cert\") pod \"ovnkube-node-2s9nz\" (UID: \"c5765452-37b1-4153-9096-778d6b33857c\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s9nz"
Jul 23 17:05:06.798873 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:05:06.798850    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 23 17:05:06.798873 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:05:06.798872    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 23 17:05:06.799011 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:05:06.798882    2577 projected.go:196] Error preparing data for projected volume kube-api-access-hpn22 for pod openshift-network-diagnostics/network-check-target-5c6b5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 23 17:05:06.799011 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:05:06.798951    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2090b02e-14f1-4224-8691-8fa4ecf7d53b-kube-api-access-hpn22 podName:2090b02e-14f1-4224-8691-8fa4ecf7d53b nodeName:}" failed. No retries permitted until 2026-07-23 17:05:07.298936393 +0000 UTC m=+3.069221897 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-hpn22" (UniqueName: "kubernetes.io/projected/2090b02e-14f1-4224-8691-8fa4ecf7d53b-kube-api-access-hpn22") pod "network-check-target-5c6b5" (UID: "2090b02e-14f1-4224-8691-8fa4ecf7d53b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 23 17:05:06.800733 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.800713    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lk7rx\" (UniqueName: \"kubernetes.io/projected/a70b8d80-c6a7-4498-ab2d-d1a7d3957ceb-kube-api-access-lk7rx\") pod \"multus-6hk44\" (UID: \"a70b8d80-c6a7-4498-ab2d-d1a7d3957ceb\") " pod="openshift-multus/multus-6hk44"
Jul 23 17:05:06.801519 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.801501    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qjcc\" (UniqueName: \"kubernetes.io/projected/88de4e3c-bd4f-44d7-a9aa-58c2c4d24549-kube-api-access-6qjcc\") pod \"iptables-alerter-ft9nr\" (UID: \"88de4e3c-bd4f-44d7-a9aa-58c2c4d24549\") " pod="openshift-network-operator/iptables-alerter-ft9nr"
Jul 23 17:05:06.801814 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.801792    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ktd2\" (UniqueName: \"kubernetes.io/projected/664f5a33-efcf-447f-8e05-a3f6ce7e4f07-kube-api-access-6ktd2\") pod \"network-metrics-daemon-mjfnh\" (UID: \"664f5a33-efcf-447f-8e05-a3f6ce7e4f07\") " pod="openshift-multus/network-metrics-daemon-mjfnh"
Jul 23 17:05:06.802836 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.802817    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvl55\" (UniqueName: \"kubernetes.io/projected/c5765452-37b1-4153-9096-778d6b33857c-kube-api-access-rvl55\") pod \"ovnkube-node-2s9nz\" (UID: \"c5765452-37b1-4153-9096-778d6b33857c\") " pod="openshift-ovn-kubernetes/ovnkube-node-2s9nz"
Jul 23 17:05:06.803502 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.803475    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvc29\" (UniqueName: \"kubernetes.io/projected/62550ab4-e316-461d-a09e-679202149bb2-kube-api-access-nvc29\") pod \"tuned-vj6t5\" (UID: \"62550ab4-e316-461d-a09e-679202149bb2\") " pod="openshift-cluster-node-tuning-operator/tuned-vj6t5"
Jul 23 17:05:06.803611 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.803579    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgbvc\" (UniqueName: \"kubernetes.io/projected/564dda72-0cfc-4e30-b04d-eddd1a9b7b30-kube-api-access-kgbvc\") pod \"node-ca-znmk5\" (UID: \"564dda72-0cfc-4e30-b04d-eddd1a9b7b30\") " pod="openshift-image-registry/node-ca-znmk5"
Jul 23 17:05:06.883903 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.883815    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-stw6g"
Jul 23 17:05:06.892745 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.892716    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p4twn"
Jul 23 17:05:06.904532 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.904507    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-b797v"
Jul 23 17:05:06.912524 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.912475    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-ft9nr"
Jul 23 17:05:06.920441 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.920420    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-pc8xz"
Jul 23 17:05:06.929161 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.929141    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-vj6t5"
Jul 23 17:05:06.937493 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.937466    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-znmk5"
Jul 23 17:05:06.945720 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.945698    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-6hk44"
Jul 23 17:05:06.952530 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:06.952503    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-2s9nz"
Jul 23 17:05:07.297081 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:07.297044    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/664f5a33-efcf-447f-8e05-a3f6ce7e4f07-metrics-certs\") pod \"network-metrics-daemon-mjfnh\" (UID: \"664f5a33-efcf-447f-8e05-a3f6ce7e4f07\") " pod="openshift-multus/network-metrics-daemon-mjfnh"
Jul 23 17:05:07.297295 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:07.297138    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/aeb10e6f-ad01-4a83-8090-049d9e919696-original-pull-secret\") pod \"global-pull-secret-syncer-5g97l\" (UID: \"aeb10e6f-ad01-4a83-8090-049d9e919696\") " pod="kube-system/global-pull-secret-syncer-5g97l"
Jul 23 17:05:07.297295 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:05:07.297205    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 23 17:05:07.297451 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:05:07.297310    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 23 17:05:07.297451 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:05:07.297331    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/664f5a33-efcf-447f-8e05-a3f6ce7e4f07-metrics-certs podName:664f5a33-efcf-447f-8e05-a3f6ce7e4f07 nodeName:}" failed. No retries permitted until 2026-07-23 17:05:08.297284675 +0000 UTC m=+4.067570191 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/664f5a33-efcf-447f-8e05-a3f6ce7e4f07-metrics-certs") pod "network-metrics-daemon-mjfnh" (UID: "664f5a33-efcf-447f-8e05-a3f6ce7e4f07") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 23 17:05:07.297451 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:05:07.297376    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aeb10e6f-ad01-4a83-8090-049d9e919696-original-pull-secret podName:aeb10e6f-ad01-4a83-8090-049d9e919696 nodeName:}" failed. No retries permitted until 2026-07-23 17:05:08.297355251 +0000 UTC m=+4.067640768 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/aeb10e6f-ad01-4a83-8090-049d9e919696-original-pull-secret") pod "global-pull-secret-syncer-5g97l" (UID: "aeb10e6f-ad01-4a83-8090-049d9e919696") : object "kube-system"/"original-pull-secret" not registered
Jul 23 17:05:07.398356 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:07.398309    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hpn22\" (UniqueName: \"kubernetes.io/projected/2090b02e-14f1-4224-8691-8fa4ecf7d53b-kube-api-access-hpn22\") pod \"network-check-target-5c6b5\" (UID: \"2090b02e-14f1-4224-8691-8fa4ecf7d53b\") " pod="openshift-network-diagnostics/network-check-target-5c6b5"
Jul 23 17:05:07.398523 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:05:07.398482    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 23 17:05:07.398523 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:05:07.398506    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 23 17:05:07.398523 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:05:07.398520    2577 projected.go:196] Error preparing data for projected volume kube-api-access-hpn22 for pod openshift-network-diagnostics/network-check-target-5c6b5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 23 17:05:07.398695 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:05:07.398587    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2090b02e-14f1-4224-8691-8fa4ecf7d53b-kube-api-access-hpn22 podName:2090b02e-14f1-4224-8691-8fa4ecf7d53b nodeName:}" failed. No retries permitted until 2026-07-23 17:05:08.398566387 +0000 UTC m=+4.168851896 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-hpn22" (UniqueName: "kubernetes.io/projected/2090b02e-14f1-4224-8691-8fa4ecf7d53b-kube-api-access-hpn22") pod "network-check-target-5c6b5" (UID: "2090b02e-14f1-4224-8691-8fa4ecf7d53b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 23 17:05:07.532976 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:07.532943    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65acd752_a782_40a8_82b9_3891d7349f33.slice/crio-9918d48a9a1d11923a56b9bba2b63191f1f65eecbc3325c208130c91bf16b8a8 WatchSource:0}: Error finding container 9918d48a9a1d11923a56b9bba2b63191f1f65eecbc3325c208130c91bf16b8a8: Status 404 returned error can't find the container with id 9918d48a9a1d11923a56b9bba2b63191f1f65eecbc3325c208130c91bf16b8a8
Jul 23 17:05:07.535733 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:07.535682    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62550ab4_e316_461d_a09e_679202149bb2.slice/crio-718643d0f335058f4ecbd7f75e8336f558283487ece317517f5ed8c83c109f5d WatchSource:0}: Error finding container 718643d0f335058f4ecbd7f75e8336f558283487ece317517f5ed8c83c109f5d: Status 404 returned error can't find the container with id 718643d0f335058f4ecbd7f75e8336f558283487ece317517f5ed8c83c109f5d
Jul 23 17:05:07.537591 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:07.537565    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71b6eaa7_70df_4f01_b0e9_6be6043acbc4.slice/crio-00dad5f76bd932daf43145630daa353bf2eae18e176f2c3cdcdffd1b91d21a5f WatchSource:0}: Error finding container 00dad5f76bd932daf43145630daa353bf2eae18e176f2c3cdcdffd1b91d21a5f: Status 404 returned error can't find the container with id 00dad5f76bd932daf43145630daa353bf2eae18e176f2c3cdcdffd1b91d21a5f
Jul 23 17:05:07.539920 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:07.539882    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a44608a_7a55_45f6_a4dc_199578a1d620.slice/crio-652ed23939905997d2d41d0cc39caab07a55633ed73b52955d7fa4e403f10f4d WatchSource:0}: Error finding container 652ed23939905997d2d41d0cc39caab07a55633ed73b52955d7fa4e403f10f4d: Status 404 returned error can't find the container with id 652ed23939905997d2d41d0cc39caab07a55633ed73b52955d7fa4e403f10f4d
Jul 23 17:05:07.540758 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:07.540731    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda70b8d80_c6a7_4498_ab2d_d1a7d3957ceb.slice/crio-12b9baf08befa5be94a3ed18497b071ebfd18815ae5a9e72e8b137ad44bff87e WatchSource:0}: Error finding container 12b9baf08befa5be94a3ed18497b071ebfd18815ae5a9e72e8b137ad44bff87e: Status 404 returned error can't find the container with id 12b9baf08befa5be94a3ed18497b071ebfd18815ae5a9e72e8b137ad44bff87e
Jul 23 17:05:07.542200 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:07.542176    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5765452_37b1_4153_9096_778d6b33857c.slice/crio-4274d0f899dcf0d458cce689a57a46877f09360a1cde615ba420c90b367bd261 WatchSource:0}: Error finding container 4274d0f899dcf0d458cce689a57a46877f09360a1cde615ba420c90b367bd261: Status 404 returned error can't find the container with id 4274d0f899dcf0d458cce689a57a46877f09360a1cde615ba420c90b367bd261
Jul 23 17:05:07.544501 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:05:07.544468    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88de4e3c_bd4f_44d7_a9aa_58c2c4d24549.slice/crio-4d9b63ec2e8efa65a08aae85c8c2dbdc8e04dee7e561fe294135ffc06b3f64b7 WatchSource:0}: Error finding container 4d9b63ec2e8efa65a08aae85c8c2dbdc8e04dee7e561fe294135ffc06b3f64b7: Status 404 returned error can't find the container with id 4d9b63ec2e8efa65a08aae85c8c2dbdc8e04dee7e561fe294135ffc06b3f64b7
Jul 23 17:05:07.661646 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:07.661577    2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-22 17:00:06 +0000 UTC" deadline="2027-12-19 04:23:46.341367381 +0000 UTC"
Jul 23 17:05:07.661646 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:07.661642    2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12323h18m38.679728691s"
Jul 23 17:05:07.759855 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:07.759817    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5c6b5"
Jul 23 17:05:07.760021 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:05:07.759940    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5c6b5" podUID="2090b02e-14f1-4224-8691-8fa4ecf7d53b"
Jul 23 17:05:07.760021 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:07.759996    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mjfnh"
Jul 23 17:05:07.760151 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:05:07.760120    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mjfnh" podUID="664f5a33-efcf-447f-8e05-a3f6ce7e4f07"
Jul 23 17:05:07.770080 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:07.770043    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-6hk44" event={"ID":"a70b8d80-c6a7-4498-ab2d-d1a7d3957ceb","Type":"ContainerStarted","Data":"12b9baf08befa5be94a3ed18497b071ebfd18815ae5a9e72e8b137ad44bff87e"}
Jul 23 17:05:07.771430 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:07.771403    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-pc8xz" event={"ID":"9a44608a-7a55-45f6-a4dc-199578a1d620","Type":"ContainerStarted","Data":"652ed23939905997d2d41d0cc39caab07a55633ed73b52955d7fa4e403f10f4d"}
Jul 23 17:05:07.772779 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:07.772755    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-vj6t5" event={"ID":"62550ab4-e316-461d-a09e-679202149bb2","Type":"ContainerStarted","Data":"718643d0f335058f4ecbd7f75e8336f558283487ece317517f5ed8c83c109f5d"}
Jul 23 17:05:07.775029 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:07.775004    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-216.ec2.internal" event={"ID":"7bd1d5c1fbf257c4a5dcb5c0b12f7f75","Type":"ContainerStarted","Data":"c5dd8b5ddf2ddaf23ba6cb2c02784a2f9abb7cacf67450c9b1250b6ffdc722ab"}
Jul 23 17:05:07.776402 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:07.776373    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-ft9nr" event={"ID":"88de4e3c-bd4f-44d7-a9aa-58c2c4d24549","Type":"ContainerStarted","Data":"4d9b63ec2e8efa65a08aae85c8c2dbdc8e04dee7e561fe294135ffc06b3f64b7"}
Jul 23 17:05:07.777783 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:07.777756    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2s9nz" event={"ID":"c5765452-37b1-4153-9096-778d6b33857c","Type":"ContainerStarted","Data":"4274d0f899dcf0d458cce689a57a46877f09360a1cde615ba420c90b367bd261"}
Jul 23 17:05:07.779234 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:07.779201    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p4twn" event={"ID":"71b6eaa7-70df-4f01-b0e9-6be6043acbc4","Type":"ContainerStarted","Data":"00dad5f76bd932daf43145630daa353bf2eae18e176f2c3cdcdffd1b91d21a5f"}
Jul 23 17:05:07.780626 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:07.780605    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-b797v" event={"ID":"65acd752-a782-40a8-82b9-3891d7349f33","Type":"ContainerStarted","Data":"9918d48a9a1d11923a56b9bba2b63191f1f65eecbc3325c208130c91bf16b8a8"}
Jul 23 17:05:07.781890 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:07.781843    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-stw6g" event={"ID":"5c929500-877b-43d7-9710-0ea27db45055","Type":"ContainerStarted","Data":"960dac3515d045a15a690451861d45ebaabdec68337d0e81ea241223c4b228ce"}
Jul 23 17:05:07.783230 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:07.783206    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-znmk5" event={"ID":"564dda72-0cfc-4e30-b04d-eddd1a9b7b30","Type":"ContainerStarted","Data":"28a3033551fb93fb4bf86c70960bddb506118e8a815d377a3a40bf18b89b69cc"}
Jul 23 17:05:07.797884 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:07.797832    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-216.ec2.internal" podStartSLOduration=1.797818655 podStartE2EDuration="1.797818655s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 17:05:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 17:05:07.797585365 +0000 UTC m=+3.567870966" watchObservedRunningTime="2026-07-23 17:05:07.797818655 +0000 UTC m=+3.568104177"
Jul 23 17:05:08.305745 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:08.305229    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/aeb10e6f-ad01-4a83-8090-049d9e919696-original-pull-secret\") pod \"global-pull-secret-syncer-5g97l\" (UID: \"aeb10e6f-ad01-4a83-8090-049d9e919696\") " pod="kube-system/global-pull-secret-syncer-5g97l"
Jul 23 17:05:08.305745 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:08.305540    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/664f5a33-efcf-447f-8e05-a3f6ce7e4f07-metrics-certs\") pod \"network-metrics-daemon-mjfnh\" (UID: \"664f5a33-efcf-447f-8e05-a3f6ce7e4f07\") " pod="openshift-multus/network-metrics-daemon-mjfnh"
Jul 23 17:05:08.305745 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:05:08.305402    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 23 17:05:08.305745 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:05:08.305637    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aeb10e6f-ad01-4a83-8090-049d9e919696-original-pull-secret podName:aeb10e6f-ad01-4a83-8090-049d9e919696 nodeName:}" failed. No retries permitted until 2026-07-23 17:05:10.3056137 +0000 UTC m=+6.075899215 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/aeb10e6f-ad01-4a83-8090-049d9e919696-original-pull-secret") pod "global-pull-secret-syncer-5g97l" (UID: "aeb10e6f-ad01-4a83-8090-049d9e919696") : object "kube-system"/"original-pull-secret" not registered
Jul 23 17:05:08.305745 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:05:08.305668    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 23 17:05:08.305745 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:05:08.305720    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/664f5a33-efcf-447f-8e05-a3f6ce7e4f07-metrics-certs podName:664f5a33-efcf-447f-8e05-a3f6ce7e4f07 nodeName:}" failed. No retries permitted until 2026-07-23 17:05:10.305703807 +0000 UTC m=+6.075989309 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/664f5a33-efcf-447f-8e05-a3f6ce7e4f07-metrics-certs") pod "network-metrics-daemon-mjfnh" (UID: "664f5a33-efcf-447f-8e05-a3f6ce7e4f07") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 23 17:05:08.406771 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:08.406695    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hpn22\" (UniqueName: \"kubernetes.io/projected/2090b02e-14f1-4224-8691-8fa4ecf7d53b-kube-api-access-hpn22\") pod \"network-check-target-5c6b5\" (UID: \"2090b02e-14f1-4224-8691-8fa4ecf7d53b\") " pod="openshift-network-diagnostics/network-check-target-5c6b5"
Jul 23 17:05:08.406903 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:05:08.406878    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 23 17:05:08.406903 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:05:08.406902    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 23 17:05:08.407017 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:05:08.406916    2577 projected.go:196] Error preparing data for projected volume kube-api-access-hpn22 for pod openshift-network-diagnostics/network-check-target-5c6b5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 23 17:05:08.407017 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:05:08.406980    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2090b02e-14f1-4224-8691-8fa4ecf7d53b-kube-api-access-hpn22 podName:2090b02e-14f1-4224-8691-8fa4ecf7d53b nodeName:}" failed. No retries permitted until 2026-07-23 17:05:10.406957962 +0000 UTC m=+6.177243473 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-hpn22" (UniqueName: "kubernetes.io/projected/2090b02e-14f1-4224-8691-8fa4ecf7d53b-kube-api-access-hpn22") pod "network-check-target-5c6b5" (UID: "2090b02e-14f1-4224-8691-8fa4ecf7d53b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 23 17:05:08.662101 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:08.662018    2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-22 17:00:06 +0000 UTC" deadline="2028-02-10 18:23:58.26693267 +0000 UTC"
Jul 23 17:05:08.662101 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:08.662055    2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13609h18m49.604882661s"
Jul 23 17:05:08.760386 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:08.760358    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5g97l"
Jul 23 17:05:08.760559 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:05:08.760495    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5g97l" podUID="aeb10e6f-ad01-4a83-8090-049d9e919696"
Jul 23 17:05:08.800358 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:08.800325    2577 generic.go:356] "Generic (PLEG): container finished" podID="28346eafc8fa850eaa3541eee81eaa75" containerID="5fb276461dc776ae43b472111767f3aab3bc0e5993586c091b1cb784335a0f70" exitCode=0
Jul 23 17:05:08.800576 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:08.800550    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-216.ec2.internal" event={"ID":"28346eafc8fa850eaa3541eee81eaa75","Type":"ContainerDied","Data":"5fb276461dc776ae43b472111767f3aab3bc0e5993586c091b1cb784335a0f70"}
Jul 23 17:05:09.760806 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:09.760086    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5c6b5"
Jul 23 17:05:09.760806 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:05:09.760220    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5c6b5" podUID="2090b02e-14f1-4224-8691-8fa4ecf7d53b"
Jul 23 17:05:09.760806 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:09.760350    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mjfnh"
Jul 23 17:05:09.760806 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:05:09.760445    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mjfnh" podUID="664f5a33-efcf-447f-8e05-a3f6ce7e4f07"
Jul 23 17:05:09.816784 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:09.816738    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-216.ec2.internal" event={"ID":"28346eafc8fa850eaa3541eee81eaa75","Type":"ContainerStarted","Data":"d7661c10d7ad5c12c4dafe69241acf5e810547d10d5bcb2405e0b1ce27a5d5c1"}
Jul 23 17:05:10.326240 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:10.326206    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/664f5a33-efcf-447f-8e05-a3f6ce7e4f07-metrics-certs\") pod \"network-metrics-daemon-mjfnh\" (UID: \"664f5a33-efcf-447f-8e05-a3f6ce7e4f07\") " pod="openshift-multus/network-metrics-daemon-mjfnh"
Jul 23 17:05:10.326446 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:10.326301    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/aeb10e6f-ad01-4a83-8090-049d9e919696-original-pull-secret\") pod \"global-pull-secret-syncer-5g97l\" (UID: \"aeb10e6f-ad01-4a83-8090-049d9e919696\") " pod="kube-system/global-pull-secret-syncer-5g97l"
Jul 23 17:05:10.326446 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:05:10.326434    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 23 17:05:10.326554 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:05:10.326498    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aeb10e6f-ad01-4a83-8090-049d9e919696-original-pull-secret podName:aeb10e6f-ad01-4a83-8090-049d9e919696 nodeName:}" failed. No retries permitted until 2026-07-23 17:05:14.326479934 +0000 UTC m=+10.096765433 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/aeb10e6f-ad01-4a83-8090-049d9e919696-original-pull-secret") pod "global-pull-secret-syncer-5g97l" (UID: "aeb10e6f-ad01-4a83-8090-049d9e919696") : object "kube-system"/"original-pull-secret" not registered
Jul 23 17:05:10.326900 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:05:10.326880    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 23 17:05:10.327011 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:05:10.326930    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/664f5a33-efcf-447f-8e05-a3f6ce7e4f07-metrics-certs podName:664f5a33-efcf-447f-8e05-a3f6ce7e4f07 nodeName:}" failed. No retries permitted until 2026-07-23 17:05:14.326917236 +0000 UTC m=+10.097202735 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/664f5a33-efcf-447f-8e05-a3f6ce7e4f07-metrics-certs") pod "network-metrics-daemon-mjfnh" (UID: "664f5a33-efcf-447f-8e05-a3f6ce7e4f07") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 23 17:05:10.427437 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:10.427398    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hpn22\" (UniqueName: \"kubernetes.io/projected/2090b02e-14f1-4224-8691-8fa4ecf7d53b-kube-api-access-hpn22\") pod \"network-check-target-5c6b5\" (UID: \"2090b02e-14f1-4224-8691-8fa4ecf7d53b\") " pod="openshift-network-diagnostics/network-check-target-5c6b5"
Jul 23 17:05:10.427637 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:05:10.427586    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 23 17:05:10.427637 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:05:10.427616    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 23 17:05:10.427637 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:05:10.427631    2577 projected.go:196] Error preparing data for projected volume kube-api-access-hpn22 for pod openshift-network-diagnostics/network-check-target-5c6b5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 23 17:05:10.427794 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:05:10.427701    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2090b02e-14f1-4224-8691-8fa4ecf7d53b-kube-api-access-hpn22 podName:2090b02e-14f1-4224-8691-8fa4ecf7d53b nodeName:}" failed. No retries permitted until 2026-07-23 17:05:14.427681066 +0000 UTC m=+10.197966581 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-hpn22" (UniqueName: "kubernetes.io/projected/2090b02e-14f1-4224-8691-8fa4ecf7d53b-kube-api-access-hpn22") pod "network-check-target-5c6b5" (UID: "2090b02e-14f1-4224-8691-8fa4ecf7d53b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 23 17:05:10.772968 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:10.772367    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5g97l"
Jul 23 17:05:10.772968 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:05:10.772538    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5g97l" podUID="aeb10e6f-ad01-4a83-8090-049d9e919696"
Jul 23 17:05:11.759661 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:11.759618    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mjfnh"
Jul 23 17:05:11.759661 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:11.759635    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5c6b5"
Jul 23 17:05:11.759882 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:05:11.759764    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mjfnh" podUID="664f5a33-efcf-447f-8e05-a3f6ce7e4f07"
Jul 23 17:05:11.759882 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:05:11.759850    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5c6b5" podUID="2090b02e-14f1-4224-8691-8fa4ecf7d53b"
Jul 23 17:05:12.763514 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:12.763477    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5g97l"
Jul 23 17:05:12.763914 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:05:12.763616    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5g97l" podUID="aeb10e6f-ad01-4a83-8090-049d9e919696"
Jul 23 17:05:13.760475 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:13.759985    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5c6b5"
Jul 23 17:05:13.760475 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:13.759999    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mjfnh"
Jul 23 17:05:13.760475 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:05:13.760112    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5c6b5" podUID="2090b02e-14f1-4224-8691-8fa4ecf7d53b"
Jul 23 17:05:13.760475 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:05:13.760150    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mjfnh" podUID="664f5a33-efcf-447f-8e05-a3f6ce7e4f07"
Jul 23 17:05:14.360984 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:14.360952    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/aeb10e6f-ad01-4a83-8090-049d9e919696-original-pull-secret\") pod \"global-pull-secret-syncer-5g97l\" (UID: \"aeb10e6f-ad01-4a83-8090-049d9e919696\") " pod="kube-system/global-pull-secret-syncer-5g97l"
Jul 23 17:05:14.361520 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:14.361016    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/664f5a33-efcf-447f-8e05-a3f6ce7e4f07-metrics-certs\") pod \"network-metrics-daemon-mjfnh\" (UID: \"664f5a33-efcf-447f-8e05-a3f6ce7e4f07\") " pod="openshift-multus/network-metrics-daemon-mjfnh"
Jul 23 17:05:14.361520 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:05:14.361123    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 23 17:05:14.361520 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:05:14.361202    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aeb10e6f-ad01-4a83-8090-049d9e919696-original-pull-secret podName:aeb10e6f-ad01-4a83-8090-049d9e919696 nodeName:}" failed. No retries permitted until 2026-07-23 17:05:22.361183083 +0000 UTC m=+18.131468592 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/aeb10e6f-ad01-4a83-8090-049d9e919696-original-pull-secret") pod "global-pull-secret-syncer-5g97l" (UID: "aeb10e6f-ad01-4a83-8090-049d9e919696") : object "kube-system"/"original-pull-secret" not registered
Jul 23 17:05:14.361520 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:05:14.361125    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 23 17:05:14.361520 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:05:14.361307    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/664f5a33-efcf-447f-8e05-a3f6ce7e4f07-metrics-certs podName:664f5a33-efcf-447f-8e05-a3f6ce7e4f07 nodeName:}" failed. No retries permitted until 2026-07-23 17:05:22.361285634 +0000 UTC m=+18.131571133 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/664f5a33-efcf-447f-8e05-a3f6ce7e4f07-metrics-certs") pod "network-metrics-daemon-mjfnh" (UID: "664f5a33-efcf-447f-8e05-a3f6ce7e4f07") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 23 17:05:14.461973 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:14.461936    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hpn22\" (UniqueName: \"kubernetes.io/projected/2090b02e-14f1-4224-8691-8fa4ecf7d53b-kube-api-access-hpn22\") pod \"network-check-target-5c6b5\" (UID: \"2090b02e-14f1-4224-8691-8fa4ecf7d53b\") " pod="openshift-network-diagnostics/network-check-target-5c6b5"
Jul 23 17:05:14.462147 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:05:14.462131    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 23 17:05:14.462206 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:05:14.462156    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 23 17:05:14.462206 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:05:14.462169    2577 projected.go:196] Error preparing data for projected volume kube-api-access-hpn22 for pod openshift-network-diagnostics/network-check-target-5c6b5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 23 17:05:14.462312 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:05:14.462237    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2090b02e-14f1-4224-8691-8fa4ecf7d53b-kube-api-access-hpn22 podName:2090b02e-14f1-4224-8691-8fa4ecf7d53b nodeName:}" failed. No retries permitted until 2026-07-23 17:05:22.462217616 +0000 UTC m=+18.232503132 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-hpn22" (UniqueName: "kubernetes.io/projected/2090b02e-14f1-4224-8691-8fa4ecf7d53b-kube-api-access-hpn22") pod "network-check-target-5c6b5" (UID: "2090b02e-14f1-4224-8691-8fa4ecf7d53b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 23 17:05:14.763498 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:14.763464    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5g97l"
Jul 23 17:05:14.763683 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:05:14.763592    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5g97l" podUID="aeb10e6f-ad01-4a83-8090-049d9e919696"
Jul 23 17:05:15.759624 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:15.759565    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5c6b5"
Jul 23 17:05:15.760092 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:05:15.759717    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5c6b5" podUID="2090b02e-14f1-4224-8691-8fa4ecf7d53b"
Jul 23 17:05:15.760092 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:15.759753    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mjfnh"
Jul 23 17:05:15.760092 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:05:15.759883    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mjfnh" podUID="664f5a33-efcf-447f-8e05-a3f6ce7e4f07"
Jul 23 17:05:16.760119 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:16.760076    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5g97l"
Jul 23 17:05:16.760617 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:05:16.760206    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5g97l" podUID="aeb10e6f-ad01-4a83-8090-049d9e919696"
Jul 23 17:05:17.760646 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:17.760569    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mjfnh"
Jul 23 17:05:17.761072 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:17.760569    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5c6b5"
Jul 23 17:05:17.761072 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:05:17.760710    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mjfnh" podUID="664f5a33-efcf-447f-8e05-a3f6ce7e4f07"
Jul 23 17:05:17.761072 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:05:17.760817    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5c6b5" podUID="2090b02e-14f1-4224-8691-8fa4ecf7d53b"
Jul 23 17:05:18.763097 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:18.763066    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5g97l"
Jul 23 17:05:18.763571 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:05:18.763180    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5g97l" podUID="aeb10e6f-ad01-4a83-8090-049d9e919696"
Jul 23 17:05:19.759746 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:19.759709    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mjfnh"
Jul 23 17:05:19.759746 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:19.759735    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5c6b5"
Jul 23 17:05:19.759942 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:05:19.759828    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mjfnh" podUID="664f5a33-efcf-447f-8e05-a3f6ce7e4f07"
Jul 23 17:05:19.760021 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:05:19.759961    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5c6b5" podUID="2090b02e-14f1-4224-8691-8fa4ecf7d53b"
Jul 23 17:05:20.759984 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:20.759943    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5g97l"
Jul 23 17:05:20.760456 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:05:20.760089    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5g97l" podUID="aeb10e6f-ad01-4a83-8090-049d9e919696"
Jul 23 17:05:21.759769 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:21.759736    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5c6b5"
Jul 23 17:05:21.759990 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:21.759776    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mjfnh"
Jul 23 17:05:21.759990 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:05:21.759852    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5c6b5" podUID="2090b02e-14f1-4224-8691-8fa4ecf7d53b"
Jul 23 17:05:21.759990 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:05:21.759975    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mjfnh" podUID="664f5a33-efcf-447f-8e05-a3f6ce7e4f07"
Jul 23 17:05:22.428621 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:22.428540    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/aeb10e6f-ad01-4a83-8090-049d9e919696-original-pull-secret\") pod \"global-pull-secret-syncer-5g97l\" (UID: \"aeb10e6f-ad01-4a83-8090-049d9e919696\") " pod="kube-system/global-pull-secret-syncer-5g97l"
Jul 23 17:05:22.428621 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:22.428605    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/664f5a33-efcf-447f-8e05-a3f6ce7e4f07-metrics-certs\") pod \"network-metrics-daemon-mjfnh\" (UID: \"664f5a33-efcf-447f-8e05-a3f6ce7e4f07\") " pod="openshift-multus/network-metrics-daemon-mjfnh"
Jul 23 17:05:22.428830 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:05:22.428698    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 23 17:05:22.428830 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:05:22.428731    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 23 17:05:22.428830 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:05:22.428776    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aeb10e6f-ad01-4a83-8090-049d9e919696-original-pull-secret podName:aeb10e6f-ad01-4a83-8090-049d9e919696 nodeName:}" failed. No retries permitted until 2026-07-23 17:05:38.428755452 +0000 UTC m=+34.199040968 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/aeb10e6f-ad01-4a83-8090-049d9e919696-original-pull-secret") pod "global-pull-secret-syncer-5g97l" (UID: "aeb10e6f-ad01-4a83-8090-049d9e919696") : object "kube-system"/"original-pull-secret" not registered
Jul 23 17:05:22.428830 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:05:22.428797    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/664f5a33-efcf-447f-8e05-a3f6ce7e4f07-metrics-certs podName:664f5a33-efcf-447f-8e05-a3f6ce7e4f07 nodeName:}" failed. No retries permitted until 2026-07-23 17:05:38.428786221 +0000 UTC m=+34.199071717 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/664f5a33-efcf-447f-8e05-a3f6ce7e4f07-metrics-certs") pod "network-metrics-daemon-mjfnh" (UID: "664f5a33-efcf-447f-8e05-a3f6ce7e4f07") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 23 17:05:22.529531 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:22.529495    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hpn22\" (UniqueName: \"kubernetes.io/projected/2090b02e-14f1-4224-8691-8fa4ecf7d53b-kube-api-access-hpn22\") pod \"network-check-target-5c6b5\" (UID: \"2090b02e-14f1-4224-8691-8fa4ecf7d53b\") " pod="openshift-network-diagnostics/network-check-target-5c6b5"
Jul 23 17:05:22.529702 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:05:22.529652    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 23 17:05:22.529702 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:05:22.529674    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 23 17:05:22.529702 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:05:22.529686    2577 projected.go:196] Error preparing data for projected volume kube-api-access-hpn22 for pod openshift-network-diagnostics/network-check-target-5c6b5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 23 17:05:22.529862 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:05:22.529744    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2090b02e-14f1-4224-8691-8fa4ecf7d53b-kube-api-access-hpn22 podName:2090b02e-14f1-4224-8691-8fa4ecf7d53b nodeName:}" failed. No retries permitted until 2026-07-23 17:05:38.529728477 +0000 UTC m=+34.300014001 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-hpn22" (UniqueName: "kubernetes.io/projected/2090b02e-14f1-4224-8691-8fa4ecf7d53b-kube-api-access-hpn22") pod "network-check-target-5c6b5" (UID: "2090b02e-14f1-4224-8691-8fa4ecf7d53b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 23 17:05:22.759807 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:22.759775    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5g97l"
Jul 23 17:05:22.759989 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:05:22.759908    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5g97l" podUID="aeb10e6f-ad01-4a83-8090-049d9e919696"
Jul 23 17:05:23.759551 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:23.759512    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5c6b5"
Jul 23 17:05:23.759966 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:23.759513    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mjfnh"
Jul 23 17:05:23.759966 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:05:23.759653    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5c6b5" podUID="2090b02e-14f1-4224-8691-8fa4ecf7d53b"
Jul 23 17:05:23.759966 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:05:23.759715    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mjfnh" podUID="664f5a33-efcf-447f-8e05-a3f6ce7e4f07"
Jul 23 17:05:24.760353 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:24.760320    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5g97l"
Jul 23 17:05:24.760855 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:05:24.760434    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5g97l" podUID="aeb10e6f-ad01-4a83-8090-049d9e919696"
Jul 23 17:05:25.759814 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:25.759778    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5c6b5"
Jul 23 17:05:25.759978 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:25.759785    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mjfnh"
Jul 23 17:05:25.759978 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:05:25.759892    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5c6b5" podUID="2090b02e-14f1-4224-8691-8fa4ecf7d53b"
Jul 23 17:05:25.760100 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:05:25.759997    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mjfnh" podUID="664f5a33-efcf-447f-8e05-a3f6ce7e4f07"
Jul 23 17:05:26.759708 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:26.759497    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5g97l"
Jul 23 17:05:26.760147 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:05:26.759816    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5g97l" podUID="aeb10e6f-ad01-4a83-8090-049d9e919696"
Jul 23 17:05:26.850361 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:26.850278    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-stw6g" event={"ID":"5c929500-877b-43d7-9710-0ea27db45055","Type":"ContainerStarted","Data":"c73ebdca87eeedada713d845b6cc4f96d16338d9e10c2da860a5285be6c85a4e"}
Jul 23 17:05:26.851855 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:26.851827    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-znmk5" event={"ID":"564dda72-0cfc-4e30-b04d-eddd1a9b7b30","Type":"ContainerStarted","Data":"4ffafed3fe62c584748404e984f4b48797da313435ae6b30ce80192817976e1a"}
Jul 23 17:05:26.853314 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:26.853286    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-6hk44" event={"ID":"a70b8d80-c6a7-4498-ab2d-d1a7d3957ceb","Type":"ContainerStarted","Data":"e77a16bb8a359d4a0e8d846d352e5fe0d307ee987793ceaead6e2e2ff1d6b40c"}
Jul 23 17:05:26.854733 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:26.854699    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-pc8xz" event={"ID":"9a44608a-7a55-45f6-a4dc-199578a1d620","Type":"ContainerStarted","Data":"d4535e298610bbd1dc3552962d2c6c254ae62b36a70cfc704a27fd87bc78983c"}
Jul 23 17:05:26.856073 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:26.856049    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-vj6t5" event={"ID":"62550ab4-e316-461d-a09e-679202149bb2","Type":"ContainerStarted","Data":"ac15113a95773e1a0d4409f2ea4671d8d73682fca798cea0809ee5542e835c08"}
Jul 23 17:05:26.857527 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:26.857506    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p4twn" event={"ID":"71b6eaa7-70df-4f01-b0e9-6be6043acbc4","Type":"ContainerStarted","Data":"49811651f2bbecb0caca39ae9cbb50782243dfbe6369f143ca94850707b2110e"}
Jul 23 17:05:26.859072 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:26.859051    2577 generic.go:356] "Generic (PLEG): container finished" podID="65acd752-a782-40a8-82b9-3891d7349f33" containerID="5acfb29325295053caf19c268310baaa0d053cff09632d6870e5e856b75dcdf2" exitCode=0
Jul 23 17:05:26.859161 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:26.859081    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-b797v" event={"ID":"65acd752-a782-40a8-82b9-3891d7349f33","Type":"ContainerDied","Data":"5acfb29325295053caf19c268310baaa0d053cff09632d6870e5e856b75dcdf2"}
Jul 23 17:05:26.865925 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:26.865877    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-stw6g" podStartSLOduration=3.025442296 podStartE2EDuration="21.865863589s" totalImagesPullingTime="18.840421293s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 17:05:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 17:05:26.865706769 +0000 UTC m=+22.635992288" watchObservedRunningTime="2026-07-23 17:05:26.865863589 +0000 UTC m=+22.636149111"
Jul 23 17:05:26.869360 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:26.866599    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-216.ec2.internal" podStartSLOduration=20.866585527 podStartE2EDuration="20.866585527s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 17:05:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 17:05:09.834186046 +0000 UTC m=+5.604471567" watchObservedRunningTime="2026-07-23 17:05:26.866585527 +0000 UTC m=+22.636871048"
Jul 23 17:05:26.899267 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:26.899220    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-pc8xz" podStartSLOduration=3.122058215 podStartE2EDuration="21.899205898s" totalImagesPullingTime="18.777147683s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 17:05:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 17:05:26.898939006 +0000 UTC m=+22.669224526" watchObservedRunningTime="2026-07-23 17:05:26.899205898 +0000 UTC m=+22.669491417"
Jul 23 17:05:26.914037 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:26.913990    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-vj6t5" podStartSLOduration=3.060767246 podStartE2EDuration="21.913972234s" totalImagesPullingTime="18.853204988s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 17:05:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 17:05:26.913690498 +0000 UTC m=+22.683976016" watchObservedRunningTime="2026-07-23 17:05:26.913972234 +0000 UTC m=+22.684257755"
Jul 23 17:05:26.932636 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:26.932596    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-6hk44" podStartSLOduration=3.005080882 podStartE2EDuration="21.932583045s" totalImagesPullingTime="18.927502163s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 17:05:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 17:05:26.932281978 +0000 UTC m=+22.702567493" watchObservedRunningTime="2026-07-23 17:05:26.932583045 +0000 UTC m=+22.702868577"
Jul 23 17:05:26.947551 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:26.947501    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-znmk5" podStartSLOduration=3.106655392 podStartE2EDuration="21.947487323s" totalImagesPullingTime="18.840831931s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 17:05:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 17:05:26.946857968 +0000 UTC m=+22.717143484" watchObservedRunningTime="2026-07-23 17:05:26.947487323 +0000 UTC m=+22.717772842"
Jul 23 17:05:27.089451 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:27.089419    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:05:27.168381 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:27.168304    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:05:27.270243 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:27.270214    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:05:27.759625 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:27.759599    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mjfnh"
Jul 23 17:05:27.759927 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:27.759639    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5c6b5"
Jul 23 17:05:27.759927 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:05:27.759711    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mjfnh" podUID="664f5a33-efcf-447f-8e05-a3f6ce7e4f07"
Jul 23 17:05:27.759927 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:05:27.759844    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5c6b5" podUID="2090b02e-14f1-4224-8691-8fa4ecf7d53b"
Jul 23 17:05:27.864062 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:27.864029    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2s9nz" event={"ID":"c5765452-37b1-4153-9096-778d6b33857c","Type":"ContainerStarted","Data":"ea0299237da81650a0fa362df329d8b25b931c7d10be3cbfd6b4fc3904847257"}
Jul 23 17:05:27.864179 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:27.864123    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2s9nz" event={"ID":"c5765452-37b1-4153-9096-778d6b33857c","Type":"ContainerStarted","Data":"4a36031086e09aff8953ca0fe8271c5a187eae8e7ed935a86dc3e4e73ffb968b"}
Jul 23 17:05:27.864230 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:27.864178    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2s9nz" event={"ID":"c5765452-37b1-4153-9096-778d6b33857c","Type":"ContainerStarted","Data":"749aaa5245578ab610b1200c7108eb0e218f319a9476b09d99cd323d29fab22c"}
Jul 23 17:05:27.972638 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:27.972606    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:05:28.074602 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:28.074517    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:05:28.249271 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:28.249198    2577 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 23 17:05:28.286459 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:28.286438    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:05:28.372307 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:28.372214    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:05:28.474543 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:28.474506    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:05:28.651275 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:28.651080    2577 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-23T17:05:28.249234723Z","UUID":"664b0d6a-c1da-4e12-8efa-cbe7d2937b69","Handler":null,"Name":"","Endpoint":""}
Jul 23 17:05:28.652880 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:28.652852    2577 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 23 17:05:28.653020 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:28.652896    2577 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 23 17:05:28.760130 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:28.760098    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5g97l"
Jul 23 17:05:28.760635 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:05:28.760221    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5g97l" podUID="aeb10e6f-ad01-4a83-8090-049d9e919696"
Jul 23 17:05:28.863485 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:28.863459    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:05:28.868568 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:28.868533    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-ft9nr" event={"ID":"88de4e3c-bd4f-44d7-a9aa-58c2c4d24549","Type":"ContainerStarted","Data":"411ca88b78448fafc20e6703dc8fa9802b600b75438148b22b707b39892fe9a2"}
Jul 23 17:05:28.873182 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:28.872979    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2s9nz" event={"ID":"c5765452-37b1-4153-9096-778d6b33857c","Type":"ContainerStarted","Data":"e34c143ce03cf9e64dea448abcc7c442643cff9d72b03cfba581e777022fbc78"}
Jul 23 17:05:28.873182 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:28.873064    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2s9nz" event={"ID":"c5765452-37b1-4153-9096-778d6b33857c","Type":"ContainerStarted","Data":"c8568e884a830ea5ed9ff7b79b14acebb29c102dd9a13a3c73c87b40138b1f15"}
Jul 23 17:05:28.873182 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:28.873125    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2s9nz" event={"ID":"c5765452-37b1-4153-9096-778d6b33857c","Type":"ContainerStarted","Data":"eea0b43a2b98ed4d22cd87ae41e807234d7b7dbd4b4b85d89b21279839ed0ccb"}
Jul 23 17:05:28.875067 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:28.875035    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p4twn" event={"ID":"71b6eaa7-70df-4f01-b0e9-6be6043acbc4","Type":"ContainerStarted","Data":"d8d9e6b11ed0ecee5687ff02773340e70c73581fe64728219644f2d191818e72"}
Jul 23 17:05:28.885418 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:28.885372    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-ft9nr" podStartSLOduration=5.035925182 podStartE2EDuration="23.885354669s" totalImagesPullingTime="18.849429487s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 17:05:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 17:05:28.885115594 +0000 UTC m=+24.655401115" watchObservedRunningTime="2026-07-23 17:05:28.885354669 +0000 UTC m=+24.655640189"
Jul 23 17:05:28.936034 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:28.935927    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:05:29.059241 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:29.059202    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:05:29.155906 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:29.155877    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-pc8xz"
Jul 23 17:05:29.156650 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:29.156628    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-pc8xz"
Jul 23 17:05:29.184047 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:29.184021    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:05:29.193576 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:29.193552    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-stw6g_5c929500-877b-43d7-9710-0ea27db45055/dns-node-resolver/0.log"
Jul 23 17:05:29.358397 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:29.358348    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:05:29.432348 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:29.432319    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:05:29.760354 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:29.760164    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mjfnh"
Jul 23 17:05:29.760785 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:29.760164    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5c6b5"
Jul 23 17:05:29.760785 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:05:29.760454    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mjfnh" podUID="664f5a33-efcf-447f-8e05-a3f6ce7e4f07"
Jul 23 17:05:29.760785 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:05:29.760549    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5c6b5" podUID="2090b02e-14f1-4224-8691-8fa4ecf7d53b"
Jul 23 17:05:29.880624 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:29.880546    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p4twn" event={"ID":"71b6eaa7-70df-4f01-b0e9-6be6043acbc4","Type":"ContainerStarted","Data":"27c035e1f68195831bf0653808bd1bd6edb3abe3e1a0c944fdaa46ba99ee8c21"}
Jul 23 17:05:29.900105 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:29.900054    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p4twn" podStartSLOduration=3.352038068 podStartE2EDuration="24.900038656s" totalImagesPullingTime="21.548000588s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 17:05:05 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 17:05:29.899683419 +0000 UTC m=+25.669968939" watchObservedRunningTime="2026-07-23 17:05:29.900038656 +0000 UTC m=+25.670324174"
Jul 23 17:05:30.191657 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:30.191630    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-znmk5_564dda72-0cfc-4e30-b04d-eddd1a9b7b30/node-ca/0.log"
Jul 23 17:05:30.194692 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:30.194668    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:05:30.611697 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:30.611617    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:05:30.715647 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:30.715613    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:05:30.759703 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:30.759673    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5g97l"
Jul 23 17:05:30.759874 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:05:30.759798    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5g97l" podUID="aeb10e6f-ad01-4a83-8090-049d9e919696"
Jul 23 17:05:30.812577 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:30.812544    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:05:31.760862 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:31.760233    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mjfnh"
Jul 23 17:05:31.760862 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:05:31.760378    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mjfnh" podUID="664f5a33-efcf-447f-8e05-a3f6ce7e4f07"
Jul 23 17:05:31.760862 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:31.760740    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5c6b5"
Jul 23 17:05:31.760862 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:05:31.760828    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5c6b5" podUID="2090b02e-14f1-4224-8691-8fa4ecf7d53b"
Jul 23 17:05:31.773935 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:31.773908    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:05:31.888365 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:31.888330    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-b797v" event={"ID":"65acd752-a782-40a8-82b9-3891d7349f33","Type":"ContainerStarted","Data":"4a73af450fe8bf222281ee4755b943a6806ecfa7742d40c0599344b0ec891cee"}
Jul 23 17:05:31.891285 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:31.891247    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2s9nz" event={"ID":"c5765452-37b1-4153-9096-778d6b33857c","Type":"ContainerStarted","Data":"8e38d2e5296cd4657bb2a315557225b2c33686b49b6982380594294d55b20d51"}
Jul 23 17:05:32.760063 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:32.760035    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5g97l"
Jul 23 17:05:32.760204 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:05:32.760124    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5g97l" podUID="aeb10e6f-ad01-4a83-8090-049d9e919696"
Jul 23 17:05:32.894465 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:32.894388    2577 generic.go:356] "Generic (PLEG): container finished" podID="65acd752-a782-40a8-82b9-3891d7349f33" containerID="4a73af450fe8bf222281ee4755b943a6806ecfa7742d40c0599344b0ec891cee" exitCode=0
Jul 23 17:05:32.894465 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:32.894438    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-b797v" event={"ID":"65acd752-a782-40a8-82b9-3891d7349f33","Type":"ContainerDied","Data":"4a73af450fe8bf222281ee4755b943a6806ecfa7742d40c0599344b0ec891cee"}
Jul 23 17:05:33.222117 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:33.222088    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:05:33.416150 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:33.416119    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:05:33.515321 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:33.515229    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:05:33.631340 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:33.631309    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:05:33.727339 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:33.727202    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:05:33.760346 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:33.760327    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mjfnh"
Jul 23 17:05:33.760441 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:33.760357    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5c6b5"
Jul 23 17:05:33.760441 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:05:33.760427    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mjfnh" podUID="664f5a33-efcf-447f-8e05-a3f6ce7e4f07"
Jul 23 17:05:33.760577 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:05:33.760558    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5c6b5" podUID="2090b02e-14f1-4224-8691-8fa4ecf7d53b"
Jul 23 17:05:33.899619 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:33.899591    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2s9nz" event={"ID":"c5765452-37b1-4153-9096-778d6b33857c","Type":"ContainerStarted","Data":"215ef92cdf27a76573dd761cad35e92d57f0920a8a2b5453a349221977b1ece7"}
Jul 23 17:05:33.900019 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:33.899821    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-2s9nz"
Jul 23 17:05:33.915175 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:33.915155    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2s9nz"
Jul 23 17:05:33.927130 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:33.927085    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-2s9nz" podStartSLOduration=9.263422532 podStartE2EDuration="28.927070668s" totalImagesPullingTime="19.663648136s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 17:05:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 17:05:33.926474535 +0000 UTC m=+29.696760054" watchObservedRunningTime="2026-07-23 17:05:33.927070668 +0000 UTC m=+29.697356186"
Jul 23 17:05:33.976705 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:33.976684    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:05:34.763412 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:34.760921    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5g97l"
Jul 23 17:05:34.763412 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:05:34.761166    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5g97l" podUID="aeb10e6f-ad01-4a83-8090-049d9e919696"
Jul 23 17:05:34.903603 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:34.903567    2577 generic.go:356] "Generic (PLEG): container finished" podID="65acd752-a782-40a8-82b9-3891d7349f33" containerID="5cdfd460a89c997b768b186679b1f4fcd96c27ff93849e2623072a9223a831a4" exitCode=0
Jul 23 17:05:34.903992 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:34.903650    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-b797v" event={"ID":"65acd752-a782-40a8-82b9-3891d7349f33","Type":"ContainerDied","Data":"5cdfd460a89c997b768b186679b1f4fcd96c27ff93849e2623072a9223a831a4"}
Jul 23 17:05:34.904061 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:34.904026    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-2s9nz"
Jul 23 17:05:34.904120 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:34.904110    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-2s9nz"
Jul 23 17:05:34.919379 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:34.919360    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2s9nz"
Jul 23 17:05:34.938635 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:34.938615    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:05:35.346639 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:35.346609    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:05:35.536862 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:35.536829    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:05:35.724472 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:35.724444    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:05:35.759720 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:35.759692    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5c6b5"
Jul 23 17:05:35.759873 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:05:35.759809    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5c6b5" podUID="2090b02e-14f1-4224-8691-8fa4ecf7d53b"
Jul 23 17:05:35.759943 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:35.759874    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mjfnh"
Jul 23 17:05:35.760025 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:05:35.760006    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mjfnh" podUID="664f5a33-efcf-447f-8e05-a3f6ce7e4f07"
Jul 23 17:05:35.941550 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:35.941521    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:05:36.046893 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:36.046834    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:05:36.242054 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:36.242028    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-pc8xz"
Jul 23 17:05:36.242179 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:36.242167    2577 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Jul 23 17:05:36.242677 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:36.242663    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-pc8xz"
Jul 23 17:05:36.759691 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:36.759660    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5g97l"
Jul 23 17:05:36.759844 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:05:36.759782    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5g97l" podUID="aeb10e6f-ad01-4a83-8090-049d9e919696"
Jul 23 17:05:36.909049 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:36.909015    2577 generic.go:356] "Generic (PLEG): container finished" podID="65acd752-a782-40a8-82b9-3891d7349f33" containerID="1cb48f6fae3254fb5f7794edf6b3e28edb26f203eeda25103b785a572adaa74c" exitCode=0
Jul 23 17:05:36.909231 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:36.909095    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-b797v" event={"ID":"65acd752-a782-40a8-82b9-3891d7349f33","Type":"ContainerDied","Data":"1cb48f6fae3254fb5f7794edf6b3e28edb26f203eeda25103b785a572adaa74c"}
Jul 23 17:05:36.918899 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:36.918879    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:05:37.000245 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:37.000217    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:05:37.327307 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:37.327244    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:05:37.533823 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:37.533783    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:05:37.752998 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:37.752962    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:05:37.760022 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:37.759990    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5c6b5"
Jul 23 17:05:37.760158 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:05:37.760099    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5c6b5" podUID="2090b02e-14f1-4224-8691-8fa4ecf7d53b"
Jul 23 17:05:37.760158 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:37.760130    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mjfnh"
Jul 23 17:05:37.760291 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:05:37.760206    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mjfnh" podUID="664f5a33-efcf-447f-8e05-a3f6ce7e4f07"
Jul 23 17:05:37.853347 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:37.853315    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:05:38.205221 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:38.205190    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:05:38.448784 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:38.448754    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/664f5a33-efcf-447f-8e05-a3f6ce7e4f07-metrics-certs\") pod \"network-metrics-daemon-mjfnh\" (UID: \"664f5a33-efcf-447f-8e05-a3f6ce7e4f07\") " pod="openshift-multus/network-metrics-daemon-mjfnh"
Jul 23 17:05:38.448948 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:38.448815    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/aeb10e6f-ad01-4a83-8090-049d9e919696-original-pull-secret\") pod \"global-pull-secret-syncer-5g97l\" (UID: \"aeb10e6f-ad01-4a83-8090-049d9e919696\") " pod="kube-system/global-pull-secret-syncer-5g97l"
Jul 23 17:05:38.448948 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:05:38.448917    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 23 17:05:38.449052 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:05:38.448997    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aeb10e6f-ad01-4a83-8090-049d9e919696-original-pull-secret podName:aeb10e6f-ad01-4a83-8090-049d9e919696 nodeName:}" failed. No retries permitted until 2026-07-23 17:06:10.448957994 +0000 UTC m=+66.219243504 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/aeb10e6f-ad01-4a83-8090-049d9e919696-original-pull-secret") pod "global-pull-secret-syncer-5g97l" (UID: "aeb10e6f-ad01-4a83-8090-049d9e919696") : object "kube-system"/"original-pull-secret" not registered
Jul 23 17:05:38.449052 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:05:38.448917    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 23 17:05:38.449137 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:05:38.449053    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/664f5a33-efcf-447f-8e05-a3f6ce7e4f07-metrics-certs podName:664f5a33-efcf-447f-8e05-a3f6ce7e4f07 nodeName:}" failed. No retries permitted until 2026-07-23 17:06:10.449040178 +0000 UTC m=+66.219325685 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/664f5a33-efcf-447f-8e05-a3f6ce7e4f07-metrics-certs") pod "network-metrics-daemon-mjfnh" (UID: "664f5a33-efcf-447f-8e05-a3f6ce7e4f07") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 23 17:05:38.550004 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:38.549926    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hpn22\" (UniqueName: \"kubernetes.io/projected/2090b02e-14f1-4224-8691-8fa4ecf7d53b-kube-api-access-hpn22\") pod \"network-check-target-5c6b5\" (UID: \"2090b02e-14f1-4224-8691-8fa4ecf7d53b\") " pod="openshift-network-diagnostics/network-check-target-5c6b5"
Jul 23 17:05:38.550168 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:05:38.550099    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 23 17:05:38.550168 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:05:38.550125    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 23 17:05:38.550168 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:05:38.550140    2577 projected.go:196] Error preparing data for projected volume kube-api-access-hpn22 for pod openshift-network-diagnostics/network-check-target-5c6b5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 23 17:05:38.550474 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:05:38.550200    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2090b02e-14f1-4224-8691-8fa4ecf7d53b-kube-api-access-hpn22 podName:2090b02e-14f1-4224-8691-8fa4ecf7d53b nodeName:}" failed. No retries permitted until 2026-07-23 17:06:10.550184625 +0000 UTC m=+66.320470147 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-hpn22" (UniqueName: "kubernetes.io/projected/2090b02e-14f1-4224-8691-8fa4ecf7d53b-kube-api-access-hpn22") pod "network-check-target-5c6b5" (UID: "2090b02e-14f1-4224-8691-8fa4ecf7d53b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 23 17:05:38.721163 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:38.721123    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:05:38.760487 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:38.760455    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5g97l"
Jul 23 17:05:38.760660 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:05:38.760592    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5g97l" podUID="aeb10e6f-ad01-4a83-8090-049d9e919696"
Jul 23 17:05:38.822991 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:38.822906    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:05:38.924261 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:38.924221    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:05:39.206352 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:39.206324    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:05:39.398924 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:39.398888    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:05:39.500753 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:39.500666    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:05:39.601558 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:39.601518    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:05:39.759856 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:39.759783    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5c6b5"
Jul 23 17:05:39.760013 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:39.759782    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mjfnh"
Jul 23 17:05:39.760013 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:05:39.759889    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5c6b5" podUID="2090b02e-14f1-4224-8691-8fa4ecf7d53b"
Jul 23 17:05:39.760013 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:05:39.759945    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mjfnh" podUID="664f5a33-efcf-447f-8e05-a3f6ce7e4f07"
Jul 23 17:05:39.772797 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:39.772762    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:05:39.933052 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:39.933019    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:05:40.759514 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:40.759477    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5g97l"
Jul 23 17:05:40.759979 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:05:40.759606    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5g97l" podUID="aeb10e6f-ad01-4a83-8090-049d9e919696"
Jul 23 17:05:41.056303 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:41.056278    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:05:41.157311 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:41.157279    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:05:41.328630 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:41.328550    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:05:41.525862 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:41.525834    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:05:41.716085 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:41.716059    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:05:41.759951 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:41.759931    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5c6b5"
Jul 23 17:05:41.760246 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:41.759931    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mjfnh"
Jul 23 17:05:41.760246 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:05:41.760025    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5c6b5" podUID="2090b02e-14f1-4224-8691-8fa4ecf7d53b"
Jul 23 17:05:41.760246 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:05:41.760095    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mjfnh" podUID="664f5a33-efcf-447f-8e05-a3f6ce7e4f07"
Jul 23 17:05:41.921840 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:41.921810    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:05:41.923056 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:41.923036    2577 generic.go:356] "Generic (PLEG): container finished" podID="65acd752-a782-40a8-82b9-3891d7349f33" containerID="d2e507445921d046895b22271a5865af066c512a355c719bf67962f83f28825c" exitCode=0
Jul 23 17:05:41.923172 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:41.923078    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-b797v" event={"ID":"65acd752-a782-40a8-82b9-3891d7349f33","Type":"ContainerDied","Data":"d2e507445921d046895b22271a5865af066c512a355c719bf67962f83f28825c"}
Jul 23 17:05:42.760363 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:42.760332    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5g97l"
Jul 23 17:05:42.760745 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:05:42.760459    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5g97l" podUID="aeb10e6f-ad01-4a83-8090-049d9e919696"
Jul 23 17:05:42.928456 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:42.928421    2577 generic.go:356] "Generic (PLEG): container finished" podID="65acd752-a782-40a8-82b9-3891d7349f33" containerID="dfdcff5a76ec3648f525f751d40a23a683d92bb08712b5ccfd7814392e055587" exitCode=0
Jul 23 17:05:42.928609 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:42.928486    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-b797v" event={"ID":"65acd752-a782-40a8-82b9-3891d7349f33","Type":"ContainerDied","Data":"dfdcff5a76ec3648f525f751d40a23a683d92bb08712b5ccfd7814392e055587"}
Jul 23 17:05:43.127906 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:43.127827    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:05:43.348649 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:43.348616    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:05:43.572357 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:43.572326    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:05:43.721137 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:43.721106    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:05:43.760192 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:43.760170    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mjfnh"
Jul 23 17:05:43.760319 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:43.760171    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5c6b5"
Jul 23 17:05:43.760319 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:05:43.760289    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mjfnh" podUID="664f5a33-efcf-447f-8e05-a3f6ce7e4f07"
Jul 23 17:05:43.760391 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:05:43.760338    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5c6b5" podUID="2090b02e-14f1-4224-8691-8fa4ecf7d53b"
Jul 23 17:05:43.933718 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:43.933644    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-b797v" event={"ID":"65acd752-a782-40a8-82b9-3891d7349f33","Type":"ContainerStarted","Data":"7f84674e767a2b33dfffd6cdc493594cd857f8d898976a64098482aea7fb0c9d"}
Jul 23 17:05:43.960987 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:43.960945    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-b797v" podStartSLOduration=8.93949445 podStartE2EDuration="38.960932447s" totalImagesPullingTime="30.021437997s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 17:05:05 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 17:05:43.960482877 +0000 UTC m=+39.730768397" watchObservedRunningTime="2026-07-23 17:05:43.960932447 +0000 UTC m=+39.731217966"
Jul 23 17:05:44.465802 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:44.465764    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:05:44.569631 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:44.569599    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:05:44.761192 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:44.761118    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5g97l"
Jul 23 17:05:44.761623 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:05:44.761199    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5g97l" podUID="aeb10e6f-ad01-4a83-8090-049d9e919696"
Jul 23 17:05:44.779515 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:44.779488    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:05:44.889814 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:44.889777    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:05:45.176887 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:45.176804    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:05:45.249352 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:45.249314    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:05:45.429342 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:45.429245    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:05:45.529726 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:45.529694    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:05:45.760205 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:45.760177    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5c6b5"
Jul 23 17:05:45.760367 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:05:45.760303    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5c6b5" podUID="2090b02e-14f1-4224-8691-8fa4ecf7d53b"
Jul 23 17:05:45.760367 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:45.760362    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mjfnh"
Jul 23 17:05:45.760474 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:05:45.760457    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mjfnh" podUID="664f5a33-efcf-447f-8e05-a3f6ce7e4f07"
Jul 23 17:05:45.787492 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:45.787467    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:05:46.013195 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:46.013122    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:05:46.091635 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:46.091604    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:05:46.759904 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:46.759878    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5g97l"
Jul 23 17:05:46.760059 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:05:46.759971    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5g97l" podUID="aeb10e6f-ad01-4a83-8090-049d9e919696"
Jul 23 17:05:46.809223 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:46.809193    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:05:46.873135 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:46.873105    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:05:47.760045 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:47.760020    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mjfnh"
Jul 23 17:05:47.760215 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:05:47.760117    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mjfnh" podUID="664f5a33-efcf-447f-8e05-a3f6ce7e4f07"
Jul 23 17:05:47.760215 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:47.760167    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5c6b5"
Jul 23 17:05:47.760314 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:05:47.760238    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5c6b5" podUID="2090b02e-14f1-4224-8691-8fa4ecf7d53b"
Jul 23 17:05:48.762931 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:48.762897    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5g97l"
Jul 23 17:05:48.763316 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:05:48.762993    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5g97l" podUID="aeb10e6f-ad01-4a83-8090-049d9e919696"
Jul 23 17:05:49.759455 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:49.759413    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5c6b5"
Jul 23 17:05:49.759620 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:05:49.759521    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5c6b5" podUID="2090b02e-14f1-4224-8691-8fa4ecf7d53b"
Jul 23 17:05:49.759620 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:49.759570    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mjfnh"
Jul 23 17:05:49.759686 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:05:49.759666    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mjfnh" podUID="664f5a33-efcf-447f-8e05-a3f6ce7e4f07"
Jul 23 17:05:50.759836 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:50.759804    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5g97l"
Jul 23 17:05:50.760382 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:05:50.759942    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5g97l" podUID="aeb10e6f-ad01-4a83-8090-049d9e919696"
Jul 23 17:05:51.623158 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:51.623127    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:05:51.759780 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:51.759757    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5c6b5"
Jul 23 17:05:51.759911 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:51.759758    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mjfnh"
Jul 23 17:05:51.760162 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:05:51.759847    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5c6b5" podUID="2090b02e-14f1-4224-8691-8fa4ecf7d53b"
Jul 23 17:05:51.760162 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:05:51.759936    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mjfnh" podUID="664f5a33-efcf-447f-8e05-a3f6ce7e4f07"
Jul 23 17:05:52.762955 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:52.762929    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5g97l"
Jul 23 17:05:52.763322 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:05:52.763019    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5g97l" podUID="aeb10e6f-ad01-4a83-8090-049d9e919696"
Jul 23 17:05:53.760085 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:53.760056    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mjfnh"
Jul 23 17:05:53.760248 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:05:53.760157    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mjfnh" podUID="664f5a33-efcf-447f-8e05-a3f6ce7e4f07"
Jul 23 17:05:53.760248 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:53.760209    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5c6b5"
Jul 23 17:05:53.760362 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:05:53.760306    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5c6b5" podUID="2090b02e-14f1-4224-8691-8fa4ecf7d53b"
Jul 23 17:05:54.761119 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:54.761088    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5g97l"
Jul 23 17:05:54.761475 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:05:54.761170    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5g97l" podUID="aeb10e6f-ad01-4a83-8090-049d9e919696"
Jul 23 17:05:55.482809 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:55.482774    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:05:55.540245 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:55.540213    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:05:55.760131 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:55.760054    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mjfnh"
Jul 23 17:05:55.760307 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:05:55.760159    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mjfnh" podUID="664f5a33-efcf-447f-8e05-a3f6ce7e4f07"
Jul 23 17:05:55.760307 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:55.760217    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5c6b5"
Jul 23 17:05:55.760432 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:05:55.760340    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5c6b5" podUID="2090b02e-14f1-4224-8691-8fa4ecf7d53b"
Jul 23 17:05:56.759873 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:56.759843    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5g97l"
Jul 23 17:05:56.760352 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:05:56.759972    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5g97l" podUID="aeb10e6f-ad01-4a83-8090-049d9e919696"
Jul 23 17:05:56.825136 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:56.825105    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:05:56.926864 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:56.926835    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:05:57.151048 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:57.150971    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:05:57.261193 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:57.261142    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:05:57.362185 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:57.362156    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:05:57.647700 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:57.647672    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:05:57.760007 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:57.759975    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mjfnh"
Jul 23 17:05:57.760478 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:05:57.760077    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mjfnh" podUID="664f5a33-efcf-447f-8e05-a3f6ce7e4f07"
Jul 23 17:05:57.760478 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:57.760132    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5c6b5"
Jul 23 17:05:57.760478 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:05:57.760210    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5c6b5" podUID="2090b02e-14f1-4224-8691-8fa4ecf7d53b"
Jul 23 17:05:58.759886 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:58.759858    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5g97l"
Jul 23 17:05:58.760043 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:05:58.759963    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5g97l" podUID="aeb10e6f-ad01-4a83-8090-049d9e919696"
Jul 23 17:05:58.808659 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:58.808628    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:05:58.881031 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:58.880999    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:05:59.759868 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:59.759840    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mjfnh"
Jul 23 17:05:59.760042 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:05:59.759840    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5c6b5"
Jul 23 17:05:59.760042 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:05:59.759944    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mjfnh" podUID="664f5a33-efcf-447f-8e05-a3f6ce7e4f07"
Jul 23 17:05:59.760042 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:05:59.759998    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5c6b5" podUID="2090b02e-14f1-4224-8691-8fa4ecf7d53b"
Jul 23 17:06:00.147406 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:00.147333    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:06:00.369618 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:00.369586    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:06:00.563895 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:00.563862    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:06:00.659390 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:00.659363    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:06:00.759865 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:00.759836    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5g97l"
Jul 23 17:06:00.760018 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:06:00.759957    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5g97l" podUID="aeb10e6f-ad01-4a83-8090-049d9e919696"
Jul 23 17:06:01.270167 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:01.270130    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-mjfnh"]
Jul 23 17:06:01.270902 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:01.270273    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mjfnh"
Jul 23 17:06:01.270902 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:06:01.270366    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mjfnh" podUID="664f5a33-efcf-447f-8e05-a3f6ce7e4f07"
Jul 23 17:06:01.271084 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:01.271062    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-5c6b5"]
Jul 23 17:06:01.271184 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:01.271164    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5c6b5"
Jul 23 17:06:01.271318 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:06:01.271290    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5c6b5" podUID="2090b02e-14f1-4224-8691-8fa4ecf7d53b"
Jul 23 17:06:01.273741 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:01.273718    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-5g97l"]
Jul 23 17:06:01.273852 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:01.273805    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5g97l"
Jul 23 17:06:01.273915 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:06:01.273900    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5g97l" podUID="aeb10e6f-ad01-4a83-8090-049d9e919696"
Jul 23 17:06:02.236529 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:02.236470    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:06:02.329206 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:02.329177    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:06:02.763010 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:02.762988    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mjfnh"
Jul 23 17:06:02.763157 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:02.763076    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5c6b5"
Jul 23 17:06:02.763157 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:06:02.763082    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mjfnh" podUID="664f5a33-efcf-447f-8e05-a3f6ce7e4f07"
Jul 23 17:06:02.763157 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:02.763101    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5g97l"
Jul 23 17:06:02.763277 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:06:02.763160    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5c6b5" podUID="2090b02e-14f1-4224-8691-8fa4ecf7d53b"
Jul 23 17:06:02.763277 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:06:02.763219    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5g97l" podUID="aeb10e6f-ad01-4a83-8090-049d9e919696"
Jul 23 17:06:04.749461 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:04.749428    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:06:04.762559 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:04.762533    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mjfnh"
Jul 23 17:06:04.762699 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:04.762609    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5g97l"
Jul 23 17:06:04.762699 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:04.762635    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5c6b5"
Jul 23 17:06:04.762699 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:06:04.762649    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mjfnh" podUID="664f5a33-efcf-447f-8e05-a3f6ce7e4f07"
Jul 23 17:06:04.762699 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:06:04.762682    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5c6b5" podUID="2090b02e-14f1-4224-8691-8fa4ecf7d53b"
Jul 23 17:06:04.762874 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:06:04.762759    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5g97l" podUID="aeb10e6f-ad01-4a83-8090-049d9e919696"
Jul 23 17:06:04.851463 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:04.851429    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:06:04.950618 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:04.950591    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:06:05.046060 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:05.045998    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:06:05.145166 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:05.145138    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:06:05.219269 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:05.219227    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:06:05.607110 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:05.607084    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-216.ec2.internal" event="NodeReady"
Jul 23 17:06:05.607296 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:05.607207    2577 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 23 17:06:05.783144 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:05.783105    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-89shm"]
Jul 23 17:06:05.785530 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:05.785508    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-89shm"
Jul 23 17:06:05.789519 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:05.789497    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-n8wmv\""
Jul 23 17:06:05.789588 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:05.789534    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 23 17:06:05.789952 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:05.789936    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 23 17:06:05.790536 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:05.790521    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 23 17:06:05.792460 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:05.792329    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 23 17:06:05.853409 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:05.853385    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0372b049-248a-43e5-84a8-198ec9bbc1cd-crio-socket\") pod \"insights-runtime-extractor-89shm\" (UID: \"0372b049-248a-43e5-84a8-198ec9bbc1cd\") " pod="openshift-insights/insights-runtime-extractor-89shm"
Jul 23 17:06:05.853409 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:05.853412    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92msw\" (UniqueName: \"kubernetes.io/projected/0372b049-248a-43e5-84a8-198ec9bbc1cd-kube-api-access-92msw\") pod \"insights-runtime-extractor-89shm\" (UID: \"0372b049-248a-43e5-84a8-198ec9bbc1cd\") " pod="openshift-insights/insights-runtime-extractor-89shm"
Jul 23 17:06:05.853576 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:05.853429    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0372b049-248a-43e5-84a8-198ec9bbc1cd-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-89shm\" (UID: \"0372b049-248a-43e5-84a8-198ec9bbc1cd\") " pod="openshift-insights/insights-runtime-extractor-89shm"
Jul 23 17:06:05.853576 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:05.853454    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0372b049-248a-43e5-84a8-198ec9bbc1cd-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-89shm\" (UID: \"0372b049-248a-43e5-84a8-198ec9bbc1cd\") " pod="openshift-insights/insights-runtime-extractor-89shm"
Jul 23 17:06:05.853576 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:05.853550    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0372b049-248a-43e5-84a8-198ec9bbc1cd-data-volume\") pod \"insights-runtime-extractor-89shm\" (UID: \"0372b049-248a-43e5-84a8-198ec9bbc1cd\") " pod="openshift-insights/insights-runtime-extractor-89shm"
Jul 23 17:06:05.856976 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:05.856936    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-89shm"]
Jul 23 17:06:05.859812 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:05.859752    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-gqsnn"]
Jul 23 17:06:05.862343 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:05.862325    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-9pw6q"]
Jul 23 17:06:05.862448 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:05.862432    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-gqsnn"
Jul 23 17:06:05.864243 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:05.864223    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-9pw6q"
Jul 23 17:06:05.868715 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:05.868696    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 23 17:06:05.871916 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:05.871899    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 23 17:06:05.872098 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:05.872074    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 23 17:06:05.872214 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:05.872181    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-j8wcn\""
Jul 23 17:06:05.872291 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:05.872274    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 23 17:06:05.872339 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:05.872301    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 23 17:06:05.873660 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:05.873643    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-5n9dp\""
Jul 23 17:06:05.878751 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:05.878733    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-gqsnn"]
Jul 23 17:06:05.886992 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:05.886972    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-9pw6q"]
Jul 23 17:06:05.954072 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:05.954044    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kpx25\" (UniqueName: \"kubernetes.io/projected/e3a03180-5756-44d1-adb3-5844e27389ff-kube-api-access-kpx25\") pod \"dns-default-9pw6q\" (UID: \"e3a03180-5756-44d1-adb3-5844e27389ff\") " pod="openshift-dns/dns-default-9pw6q"
Jul 23 17:06:05.954186 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:05.954080    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0372b049-248a-43e5-84a8-198ec9bbc1cd-data-volume\") pod \"insights-runtime-extractor-89shm\" (UID: \"0372b049-248a-43e5-84a8-198ec9bbc1cd\") " pod="openshift-insights/insights-runtime-extractor-89shm"
Jul 23 17:06:05.954186 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:05.954099    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3a03180-5756-44d1-adb3-5844e27389ff-tmp-dir\") pod \"dns-default-9pw6q\" (UID: \"e3a03180-5756-44d1-adb3-5844e27389ff\") " pod="openshift-dns/dns-default-9pw6q"
Jul 23 17:06:05.954186 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:05.954119    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twc59\" (UniqueName: \"kubernetes.io/projected/1f32f423-161d-444d-9698-040a50020b5c-kube-api-access-twc59\") pod \"ingress-canary-gqsnn\" (UID: \"1f32f423-161d-444d-9698-040a50020b5c\") " pod="openshift-ingress-canary/ingress-canary-gqsnn"
Jul 23 17:06:05.954360 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:05.954295    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e3a03180-5756-44d1-adb3-5844e27389ff-config-volume\") pod \"dns-default-9pw6q\" (UID: \"e3a03180-5756-44d1-adb3-5844e27389ff\") " pod="openshift-dns/dns-default-9pw6q"
Jul 23 17:06:05.954360 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:05.954323    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1f32f423-161d-444d-9698-040a50020b5c-cert\") pod \"ingress-canary-gqsnn\" (UID: \"1f32f423-161d-444d-9698-040a50020b5c\") " pod="openshift-ingress-canary/ingress-canary-gqsnn"
Jul 23 17:06:05.954430 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:05.954360    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0372b049-248a-43e5-84a8-198ec9bbc1cd-crio-socket\") pod \"insights-runtime-extractor-89shm\" (UID: \"0372b049-248a-43e5-84a8-198ec9bbc1cd\") " pod="openshift-insights/insights-runtime-extractor-89shm"
Jul 23 17:06:05.954430 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:05.954377    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92msw\" (UniqueName: \"kubernetes.io/projected/0372b049-248a-43e5-84a8-198ec9bbc1cd-kube-api-access-92msw\") pod \"insights-runtime-extractor-89shm\" (UID: \"0372b049-248a-43e5-84a8-198ec9bbc1cd\") " pod="openshift-insights/insights-runtime-extractor-89shm"
Jul 23 17:06:05.954430 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:05.954397    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0372b049-248a-43e5-84a8-198ec9bbc1cd-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-89shm\" (UID: \"0372b049-248a-43e5-84a8-198ec9bbc1cd\") " pod="openshift-insights/insights-runtime-extractor-89shm"
Jul 23 17:06:05.954430 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:05.954423    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0372b049-248a-43e5-84a8-198ec9bbc1cd-crio-socket\") pod \"insights-runtime-extractor-89shm\" (UID: \"0372b049-248a-43e5-84a8-198ec9bbc1cd\") " pod="openshift-insights/insights-runtime-extractor-89shm"
Jul 23 17:06:05.954590 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:05.954425    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0372b049-248a-43e5-84a8-198ec9bbc1cd-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-89shm\" (UID: \"0372b049-248a-43e5-84a8-198ec9bbc1cd\") " pod="openshift-insights/insights-runtime-extractor-89shm"
Jul 23 17:06:05.954590 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:05.954502    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0372b049-248a-43e5-84a8-198ec9bbc1cd-data-volume\") pod \"insights-runtime-extractor-89shm\" (UID: \"0372b049-248a-43e5-84a8-198ec9bbc1cd\") " pod="openshift-insights/insights-runtime-extractor-89shm"
Jul 23 17:06:05.954590 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:05.954509    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e3a03180-5756-44d1-adb3-5844e27389ff-metrics-tls\") pod \"dns-default-9pw6q\" (UID: \"e3a03180-5756-44d1-adb3-5844e27389ff\") " pod="openshift-dns/dns-default-9pw6q"
Jul 23 17:06:05.954935 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:05.954916    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0372b049-248a-43e5-84a8-198ec9bbc1cd-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-89shm\" (UID: \"0372b049-248a-43e5-84a8-198ec9bbc1cd\") " pod="openshift-insights/insights-runtime-extractor-89shm"
Jul 23 17:06:05.958481 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:05.958461    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0372b049-248a-43e5-84a8-198ec9bbc1cd-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-89shm\" (UID: \"0372b049-248a-43e5-84a8-198ec9bbc1cd\") " pod="openshift-insights/insights-runtime-extractor-89shm"
Jul 23 17:06:05.981584 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:05.981563    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-92msw\" (UniqueName: \"kubernetes.io/projected/0372b049-248a-43e5-84a8-198ec9bbc1cd-kube-api-access-92msw\") pod \"insights-runtime-extractor-89shm\" (UID: \"0372b049-248a-43e5-84a8-198ec9bbc1cd\") " pod="openshift-insights/insights-runtime-extractor-89shm"
Jul 23 17:06:06.054997 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:06.054972    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kpx25\" (UniqueName: \"kubernetes.io/projected/e3a03180-5756-44d1-adb3-5844e27389ff-kube-api-access-kpx25\") pod \"dns-default-9pw6q\" (UID: \"e3a03180-5756-44d1-adb3-5844e27389ff\") " pod="openshift-dns/dns-default-9pw6q"
Jul 23 17:06:06.055100 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:06.055006    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3a03180-5756-44d1-adb3-5844e27389ff-tmp-dir\") pod \"dns-default-9pw6q\" (UID: \"e3a03180-5756-44d1-adb3-5844e27389ff\") " pod="openshift-dns/dns-default-9pw6q"
Jul 23 17:06:06.055100 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:06.055031    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twc59\" (UniqueName: \"kubernetes.io/projected/1f32f423-161d-444d-9698-040a50020b5c-kube-api-access-twc59\") pod \"ingress-canary-gqsnn\" (UID: \"1f32f423-161d-444d-9698-040a50020b5c\") " pod="openshift-ingress-canary/ingress-canary-gqsnn"
Jul 23 17:06:06.055194 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:06.055097    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e3a03180-5756-44d1-adb3-5844e27389ff-config-volume\") pod \"dns-default-9pw6q\" (UID: \"e3a03180-5756-44d1-adb3-5844e27389ff\") " pod="openshift-dns/dns-default-9pw6q"
Jul 23 17:06:06.055194 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:06.055121    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1f32f423-161d-444d-9698-040a50020b5c-cert\") pod \"ingress-canary-gqsnn\" (UID: \"1f32f423-161d-444d-9698-040a50020b5c\") " pod="openshift-ingress-canary/ingress-canary-gqsnn"
Jul 23 17:06:06.055194 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:06.055171    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e3a03180-5756-44d1-adb3-5844e27389ff-metrics-tls\") pod \"dns-default-9pw6q\" (UID: \"e3a03180-5756-44d1-adb3-5844e27389ff\") " pod="openshift-dns/dns-default-9pw6q"
Jul 23 17:06:06.055376 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:06.055343    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3a03180-5756-44d1-adb3-5844e27389ff-tmp-dir\") pod \"dns-default-9pw6q\" (UID: \"e3a03180-5756-44d1-adb3-5844e27389ff\") " pod="openshift-dns/dns-default-9pw6q"
Jul 23 17:06:06.055632 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:06.055611    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e3a03180-5756-44d1-adb3-5844e27389ff-config-volume\") pod \"dns-default-9pw6q\" (UID: \"e3a03180-5756-44d1-adb3-5844e27389ff\") " pod="openshift-dns/dns-default-9pw6q"
Jul 23 17:06:06.057278 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:06.057249    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e3a03180-5756-44d1-adb3-5844e27389ff-metrics-tls\") pod \"dns-default-9pw6q\" (UID: \"e3a03180-5756-44d1-adb3-5844e27389ff\") " pod="openshift-dns/dns-default-9pw6q"
Jul 23 17:06:06.057410 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:06.057396    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1f32f423-161d-444d-9698-040a50020b5c-cert\") pod \"ingress-canary-gqsnn\" (UID: \"1f32f423-161d-444d-9698-040a50020b5c\") " pod="openshift-ingress-canary/ingress-canary-gqsnn"
Jul 23 17:06:06.078560 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:06.078538    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kpx25\" (UniqueName: \"kubernetes.io/projected/e3a03180-5756-44d1-adb3-5844e27389ff-kube-api-access-kpx25\") pod \"dns-default-9pw6q\" (UID: \"e3a03180-5756-44d1-adb3-5844e27389ff\") " pod="openshift-dns/dns-default-9pw6q"
Jul 23 17:06:06.079769 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:06.079751    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-twc59\" (UniqueName: \"kubernetes.io/projected/1f32f423-161d-444d-9698-040a50020b5c-kube-api-access-twc59\") pod \"ingress-canary-gqsnn\" (UID: \"1f32f423-161d-444d-9698-040a50020b5c\") " pod="openshift-ingress-canary/ingress-canary-gqsnn"
Jul 23 17:06:06.095207 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:06.095191    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-89shm"
Jul 23 17:06:06.114102 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:06.114035    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:06:06.171443 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:06.171299    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-gqsnn"
Jul 23 17:06:06.175736 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:06.175708    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-9pw6q"
Jul 23 17:06:06.251874 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:06.250023    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-89shm"]
Jul 23 17:06:06.255206 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:06:06.255146    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0372b049_248a_43e5_84a8_198ec9bbc1cd.slice/crio-def3c1c0050e3a7937689479170f69c845d183b5d43737bb054505e615588713 WatchSource:0}: Error finding container def3c1c0050e3a7937689479170f69c845d183b5d43737bb054505e615588713: Status 404 returned error can't find the container with id def3c1c0050e3a7937689479170f69c845d183b5d43737bb054505e615588713
Jul 23 17:06:06.306597 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:06.306566    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:06:06.353639 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:06.353612    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-gqsnn"]
Jul 23 17:06:06.355610 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:06.355482    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-9pw6q"]
Jul 23 17:06:06.357486 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:06:06.357465    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f32f423_161d_444d_9698_040a50020b5c.slice/crio-cc3c50d5e3542593d8a7f008dc53b94043ba024698d8dcee9ba765e454725f48 WatchSource:0}: Error finding container cc3c50d5e3542593d8a7f008dc53b94043ba024698d8dcee9ba765e454725f48: Status 404 returned error can't find the container with id cc3c50d5e3542593d8a7f008dc53b94043ba024698d8dcee9ba765e454725f48
Jul 23 17:06:06.358479 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:06:06.358459    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3a03180_5756_44d1_adb3_5844e27389ff.slice/crio-5c91fb7f53a9ae46ad5df52580d8a02d6385b110dc214755c49b4543dc1af4d7 WatchSource:0}: Error finding container 5c91fb7f53a9ae46ad5df52580d8a02d6385b110dc214755c49b4543dc1af4d7: Status 404 returned error can't find the container with id 5c91fb7f53a9ae46ad5df52580d8a02d6385b110dc214755c49b4543dc1af4d7
Jul 23 17:06:06.423912 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:06.423886    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:06:06.759750 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:06.759708    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5c6b5"
Jul 23 17:06:06.760150 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:06.760119    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5g97l"
Jul 23 17:06:06.760955 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:06.760936    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mjfnh"
Jul 23 17:06:06.766460 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:06.766272    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 23 17:06:06.767996 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:06.767971    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-fr6xr\""
Jul 23 17:06:06.768269 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:06.768242    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 23 17:06:06.768467 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:06.768447    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 23 17:06:06.768711 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:06.768687    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 23 17:06:06.769119 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:06.768885    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-cgv7q\""
Jul 23 17:06:06.932435 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:06.930893    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2s9nz"
Jul 23 17:06:06.988157 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:06.988075    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-9pw6q" event={"ID":"e3a03180-5756-44d1-adb3-5844e27389ff","Type":"ContainerStarted","Data":"5c91fb7f53a9ae46ad5df52580d8a02d6385b110dc214755c49b4543dc1af4d7"}
Jul 23 17:06:06.992836 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:06.992755    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-gqsnn" event={"ID":"1f32f423-161d-444d-9698-040a50020b5c","Type":"ContainerStarted","Data":"cc3c50d5e3542593d8a7f008dc53b94043ba024698d8dcee9ba765e454725f48"}
Jul 23 17:06:06.995352 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:06.995288    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-89shm" event={"ID":"0372b049-248a-43e5-84a8-198ec9bbc1cd","Type":"ContainerStarted","Data":"def3c1c0050e3a7937689479170f69c845d183b5d43737bb054505e615588713"}
Jul 23 17:06:07.102932 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:07.102776    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:06:07.184346 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:07.184267    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:06:07.480496 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:07.480458    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:06:07.579614 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:07.579580    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:06:08.300364 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:08.300271    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:06:08.523067 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:08.523026    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:06:08.685529 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:08.685434    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:06:08.799919 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:08.799884    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:06:08.876728 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:08.876689    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:06:10.490708 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:10.490670    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/aeb10e6f-ad01-4a83-8090-049d9e919696-original-pull-secret\") pod \"global-pull-secret-syncer-5g97l\" (UID: \"aeb10e6f-ad01-4a83-8090-049d9e919696\") " pod="kube-system/global-pull-secret-syncer-5g97l"
Jul 23 17:06:10.491162 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:10.490729    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/664f5a33-efcf-447f-8e05-a3f6ce7e4f07-metrics-certs\") pod \"network-metrics-daemon-mjfnh\" (UID: \"664f5a33-efcf-447f-8e05-a3f6ce7e4f07\") " pod="openshift-multus/network-metrics-daemon-mjfnh"
Jul 23 17:06:10.493913 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:10.493883    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/aeb10e6f-ad01-4a83-8090-049d9e919696-original-pull-secret\") pod \"global-pull-secret-syncer-5g97l\" (UID: \"aeb10e6f-ad01-4a83-8090-049d9e919696\") " pod="kube-system/global-pull-secret-syncer-5g97l"
Jul 23 17:06:10.494039 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:10.493943    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/664f5a33-efcf-447f-8e05-a3f6ce7e4f07-metrics-certs\") pod \"network-metrics-daemon-mjfnh\" (UID: \"664f5a33-efcf-447f-8e05-a3f6ce7e4f07\") " pod="openshift-multus/network-metrics-daemon-mjfnh"
Jul 23 17:06:10.591530 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:10.591493    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hpn22\" (UniqueName: \"kubernetes.io/projected/2090b02e-14f1-4224-8691-8fa4ecf7d53b-kube-api-access-hpn22\") pod \"network-check-target-5c6b5\" (UID: \"2090b02e-14f1-4224-8691-8fa4ecf7d53b\") " pod="openshift-network-diagnostics/network-check-target-5c6b5"
Jul 23 17:06:10.594455 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:10.594426    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hpn22\" (UniqueName: \"kubernetes.io/projected/2090b02e-14f1-4224-8691-8fa4ecf7d53b-kube-api-access-hpn22\") pod \"network-check-target-5c6b5\" (UID: \"2090b02e-14f1-4224-8691-8fa4ecf7d53b\") " pod="openshift-network-diagnostics/network-check-target-5c6b5"
Jul 23 17:06:10.677317 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:10.677281    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5c6b5"
Jul 23 17:06:10.686446 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:10.686415    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5g97l"
Jul 23 17:06:10.692862 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:10.692839    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mjfnh"
Jul 23 17:06:11.009103 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:11.009069    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:06:11.137388 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:11.137356    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:06:11.570816 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:11.570785    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:06:11.656671 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:11.656634    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:06:12.019878 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:12.019546    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-gqsnn" event={"ID":"1f32f423-161d-444d-9698-040a50020b5c","Type":"ContainerStarted","Data":"415af60fc0a1edfeb4f2e8859a7c0e78f9b4d24edb3c9196f8f96ddce4fa8d6c"}
Jul 23 17:06:12.045325 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:12.045299    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-mjfnh"]
Jul 23 17:06:12.049295 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:06:12.049242    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod664f5a33_efcf_447f_8e05_a3f6ce7e4f07.slice/crio-0b6755b1fd6db6962c7cae77c7d17c77d33a41f0d368a3a78cb44efbcaeededb WatchSource:0}: Error finding container 0b6755b1fd6db6962c7cae77c7d17c77d33a41f0d368a3a78cb44efbcaeededb: Status 404 returned error can't find the container with id 0b6755b1fd6db6962c7cae77c7d17c77d33a41f0d368a3a78cb44efbcaeededb
Jul 23 17:06:12.056627 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:06:12.056595    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaeb10e6f_ad01_4a83_8090_049d9e919696.slice/crio-fcfb576ddc13f82dcd10d32780bd184a4fa37d67188d2210e77402b6d7a8b70c WatchSource:0}: Error finding container fcfb576ddc13f82dcd10d32780bd184a4fa37d67188d2210e77402b6d7a8b70c: Status 404 returned error can't find the container with id fcfb576ddc13f82dcd10d32780bd184a4fa37d67188d2210e77402b6d7a8b70c
Jul 23 17:06:12.061277 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:12.060329    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-5g97l"]
Jul 23 17:06:12.069950 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:12.069497    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-gqsnn" podStartSLOduration=1.574098619 podStartE2EDuration="7.069475121s" totalImagesPullingTime="5.495376502s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 17:06:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 17:06:12.063044859 +0000 UTC m=+67.833330379" watchObservedRunningTime="2026-07-23 17:06:12.069475121 +0000 UTC m=+67.839760641"
Jul 23 17:06:12.069950 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:12.069897    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-5c6b5"]
Jul 23 17:06:12.075426 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:06:12.075370    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2090b02e_14f1_4224_8691_8fa4ecf7d53b.slice/crio-f060800bf31b2fd46fec78051328b12e6d01bdd7e14c6f0521247cde3d915309 WatchSource:0}: Error finding container f060800bf31b2fd46fec78051328b12e6d01bdd7e14c6f0521247cde3d915309: Status 404 returned error can't find the container with id f060800bf31b2fd46fec78051328b12e6d01bdd7e14c6f0521247cde3d915309
Jul 23 17:06:13.024927 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:13.024883    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-5c6b5" event={"ID":"2090b02e-14f1-4224-8691-8fa4ecf7d53b","Type":"ContainerStarted","Data":"f060800bf31b2fd46fec78051328b12e6d01bdd7e14c6f0521247cde3d915309"}
Jul 23 17:06:13.027069 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:13.027038    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-mjfnh" event={"ID":"664f5a33-efcf-447f-8e05-a3f6ce7e4f07","Type":"ContainerStarted","Data":"0b6755b1fd6db6962c7cae77c7d17c77d33a41f0d368a3a78cb44efbcaeededb"}
Jul 23 17:06:13.028883 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:13.028857    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-89shm" event={"ID":"0372b049-248a-43e5-84a8-198ec9bbc1cd","Type":"ContainerStarted","Data":"4ebfc094da2c2562fecd8a1137e636229e0afc596b0488e48ff5a31b5fad3a10"}
Jul 23 17:06:13.031085 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:13.031058    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-9pw6q" event={"ID":"e3a03180-5756-44d1-adb3-5844e27389ff","Type":"ContainerStarted","Data":"0926c226465cd6b4fd59f3c46002647abd4bfffdab44acb92c9c77e7886ab823"}
Jul 23 17:06:13.031196 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:13.031109    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-9pw6q" event={"ID":"e3a03180-5756-44d1-adb3-5844e27389ff","Type":"ContainerStarted","Data":"3f7bc8977375c2f963d20cb64cf65c86173d2d02381fa8a81c41486ec4260e0a"}
Jul 23 17:06:13.031958 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:13.031938    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-9pw6q"
Jul 23 17:06:13.034118 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:13.034096    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-5g97l" event={"ID":"aeb10e6f-ad01-4a83-8090-049d9e919696","Type":"ContainerStarted","Data":"fcfb576ddc13f82dcd10d32780bd184a4fa37d67188d2210e77402b6d7a8b70c"}
Jul 23 17:06:13.066637 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:13.058898    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-9pw6q" podStartSLOduration=2.886346648 podStartE2EDuration="8.058877349s" totalImagesPullingTime="5.172530701s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 17:06:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 17:06:13.057113238 +0000 UTC m=+68.827398759" watchObservedRunningTime="2026-07-23 17:06:13.058877349 +0000 UTC m=+68.829162847"
Jul 23 17:06:13.369683 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:13.369566    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:06:13.418129 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:13.417164    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-5945b5cfd-c8579"]
Jul 23 17:06:13.420554 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:13.419495    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-5945b5cfd-c8579"
Jul 23 17:06:13.422848 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:13.422679    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 23 17:06:13.423463 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:13.423442    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 23 17:06:13.423687 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:13.423668    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-tvjng\""
Jul 23 17:06:13.424028 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:13.424010    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 23 17:06:13.424222 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:13.424184    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Jul 23 17:06:13.426447 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:13.424395    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Jul 23 17:06:13.437414 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:13.436305    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-5945b5cfd-c8579"]
Jul 23 17:06:13.456869 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:13.456783    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:06:13.515986 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:13.515689    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/3d3ac6d1-2506-4b3e-9336-4f8bdc1a87c2-prometheus-operator-tls\") pod \"prometheus-operator-5945b5cfd-c8579\" (UID: \"3d3ac6d1-2506-4b3e-9336-4f8bdc1a87c2\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-c8579"
Jul 23 17:06:13.515986 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:13.515807    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3d3ac6d1-2506-4b3e-9336-4f8bdc1a87c2-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-5945b5cfd-c8579\" (UID: \"3d3ac6d1-2506-4b3e-9336-4f8bdc1a87c2\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-c8579"
Jul 23 17:06:13.515986 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:13.515839    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3d3ac6d1-2506-4b3e-9336-4f8bdc1a87c2-metrics-client-ca\") pod \"prometheus-operator-5945b5cfd-c8579\" (UID: \"3d3ac6d1-2506-4b3e-9336-4f8bdc1a87c2\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-c8579"
Jul 23 17:06:13.515986 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:13.515864    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mkjcc\" (UniqueName: \"kubernetes.io/projected/3d3ac6d1-2506-4b3e-9336-4f8bdc1a87c2-kube-api-access-mkjcc\") pod \"prometheus-operator-5945b5cfd-c8579\" (UID: \"3d3ac6d1-2506-4b3e-9336-4f8bdc1a87c2\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-c8579"
Jul 23 17:06:13.617681 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:13.616279    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3d3ac6d1-2506-4b3e-9336-4f8bdc1a87c2-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-5945b5cfd-c8579\" (UID: \"3d3ac6d1-2506-4b3e-9336-4f8bdc1a87c2\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-c8579"
Jul 23 17:06:13.617681 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:13.616328    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3d3ac6d1-2506-4b3e-9336-4f8bdc1a87c2-metrics-client-ca\") pod \"prometheus-operator-5945b5cfd-c8579\" (UID: \"3d3ac6d1-2506-4b3e-9336-4f8bdc1a87c2\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-c8579"
Jul 23 17:06:13.617681 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:13.616357    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkjcc\" (UniqueName: \"kubernetes.io/projected/3d3ac6d1-2506-4b3e-9336-4f8bdc1a87c2-kube-api-access-mkjcc\") pod \"prometheus-operator-5945b5cfd-c8579\" (UID: \"3d3ac6d1-2506-4b3e-9336-4f8bdc1a87c2\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-c8579"
Jul 23 17:06:13.617681 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:13.616394    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/3d3ac6d1-2506-4b3e-9336-4f8bdc1a87c2-prometheus-operator-tls\") pod \"prometheus-operator-5945b5cfd-c8579\" (UID: \"3d3ac6d1-2506-4b3e-9336-4f8bdc1a87c2\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-c8579"
Jul 23 17:06:13.618766 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:13.617198    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3d3ac6d1-2506-4b3e-9336-4f8bdc1a87c2-metrics-client-ca\") pod \"prometheus-operator-5945b5cfd-c8579\" (UID: \"3d3ac6d1-2506-4b3e-9336-4f8bdc1a87c2\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-c8579"
Jul 23 17:06:13.618766 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:06:13.618463    2577 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found
Jul 23 17:06:13.618766 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:06:13.618531    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3d3ac6d1-2506-4b3e-9336-4f8bdc1a87c2-prometheus-operator-tls podName:3d3ac6d1-2506-4b3e-9336-4f8bdc1a87c2 nodeName:}" failed. No retries permitted until 2026-07-23 17:06:14.118511301 +0000 UTC m=+69.888796811 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/3d3ac6d1-2506-4b3e-9336-4f8bdc1a87c2-prometheus-operator-tls") pod "prometheus-operator-5945b5cfd-c8579" (UID: "3d3ac6d1-2506-4b3e-9336-4f8bdc1a87c2") : secret "prometheus-operator-tls" not found
Jul 23 17:06:13.620292 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:13.620221    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3d3ac6d1-2506-4b3e-9336-4f8bdc1a87c2-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-5945b5cfd-c8579\" (UID: \"3d3ac6d1-2506-4b3e-9336-4f8bdc1a87c2\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-c8579"
Jul 23 17:06:13.628293 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:13.628227    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkjcc\" (UniqueName: \"kubernetes.io/projected/3d3ac6d1-2506-4b3e-9336-4f8bdc1a87c2-kube-api-access-mkjcc\") pod \"prometheus-operator-5945b5cfd-c8579\" (UID: \"3d3ac6d1-2506-4b3e-9336-4f8bdc1a87c2\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-c8579"
Jul 23 17:06:13.869735 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:13.869702    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:06:13.970011 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:13.969937    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:06:14.048132 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:14.048097    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-mjfnh" event={"ID":"664f5a33-efcf-447f-8e05-a3f6ce7e4f07","Type":"ContainerStarted","Data":"55befbc193295bc7beafc8b0bafab9ef67e737f99f19ebab035bd3653d751e16"}
Jul 23 17:06:14.122519 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:14.122002    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/3d3ac6d1-2506-4b3e-9336-4f8bdc1a87c2-prometheus-operator-tls\") pod \"prometheus-operator-5945b5cfd-c8579\" (UID: \"3d3ac6d1-2506-4b3e-9336-4f8bdc1a87c2\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-c8579"
Jul 23 17:06:14.129341 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:14.129310    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/3d3ac6d1-2506-4b3e-9336-4f8bdc1a87c2-prometheus-operator-tls\") pod \"prometheus-operator-5945b5cfd-c8579\" (UID: \"3d3ac6d1-2506-4b3e-9336-4f8bdc1a87c2\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-c8579"
Jul 23 17:06:14.336573 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:14.336536    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-5945b5cfd-c8579"
Jul 23 17:06:14.510863 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:14.510392    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-5945b5cfd-c8579"]
Jul 23 17:06:14.514601 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:06:14.514570    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d3ac6d1_2506_4b3e_9336_4f8bdc1a87c2.slice/crio-34f48a0cf07285b72895e1694682a7c41df371f4d84cd2657d7fd81722aaf4ec WatchSource:0}: Error finding container 34f48a0cf07285b72895e1694682a7c41df371f4d84cd2657d7fd81722aaf4ec: Status 404 returned error can't find the container with id 34f48a0cf07285b72895e1694682a7c41df371f4d84cd2657d7fd81722aaf4ec
Jul 23 17:06:15.057449 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:15.057407    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-5945b5cfd-c8579" event={"ID":"3d3ac6d1-2506-4b3e-9336-4f8bdc1a87c2","Type":"ContainerStarted","Data":"34f48a0cf07285b72895e1694682a7c41df371f4d84cd2657d7fd81722aaf4ec"}
Jul 23 17:06:15.059631 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:15.059575    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-mjfnh" event={"ID":"664f5a33-efcf-447f-8e05-a3f6ce7e4f07","Type":"ContainerStarted","Data":"623371f26a9bae9db74df143b25230c8851c3c8f328f53c128892f4a96b2526f"}
Jul 23 17:06:15.076635 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:15.076582    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-mjfnh" podStartSLOduration=68.391510407 podStartE2EDuration="1m10.076566292s" totalImagesPullingTime="1.685055885s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 17:05:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 17:06:15.076215286 +0000 UTC m=+70.846500806" watchObservedRunningTime="2026-07-23 17:06:15.076566292 +0000 UTC m=+70.846851812"
Jul 23 17:06:15.183031 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:15.182964    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:06:15.253623 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:15.253578    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:06:15.342399 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:15.342322    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:06:15.570986 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:15.570939    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:06:15.641815 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:15.641725    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:06:16.836989 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:16.836952    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:06:16.901689 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:16.901652    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:06:18.101916 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:18.101879    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:06:18.230770 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:18.230729    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:06:18.316491 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:18.316434    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:06:19.680277 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:19.680176    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:06:20.076423 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:20.076383    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-5g97l" event={"ID":"aeb10e6f-ad01-4a83-8090-049d9e919696","Type":"ContainerStarted","Data":"cd9e55ccb1b2d84a8e1c4078da54c4170d534e2c38a598831c95a34aab448f7a"}
Jul 23 17:06:20.078387 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:20.078355    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-5945b5cfd-c8579" event={"ID":"3d3ac6d1-2506-4b3e-9336-4f8bdc1a87c2","Type":"ContainerStarted","Data":"eca324ee32880863b384d9ddd880917c5f0ea62e8b9560b168146b0da6243cda"}
Jul 23 17:06:20.078533 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:20.078414    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-5945b5cfd-c8579" event={"ID":"3d3ac6d1-2506-4b3e-9336-4f8bdc1a87c2","Type":"ContainerStarted","Data":"667bc38a7cac6e6a0037c85ddce4ced92ef1917d9d68cdfaf2514ab6b6ca3531"}
Jul 23 17:06:20.080007 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:20.079979    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-5c6b5" event={"ID":"2090b02e-14f1-4224-8691-8fa4ecf7d53b","Type":"ContainerStarted","Data":"01cb54250ea9012f73765089efd31b91cce519744bcdc73744f2283ecf377596"}
Jul 23 17:06:20.080146 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:20.080069    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-5c6b5"
Jul 23 17:06:20.082073 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:20.082026    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-89shm" event={"ID":"0372b049-248a-43e5-84a8-198ec9bbc1cd","Type":"ContainerStarted","Data":"3177c431dce4285e63ac80cb6bcc07d72fc958ff5fe09eb9ff7ca460d5d7a07e"}
Jul 23 17:06:20.093128 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:20.092981    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-5g97l" podStartSLOduration=67.800543735 podStartE2EDuration="1m15.09296305s" totalImagesPullingTime="7.292419315s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 17:05:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 17:06:20.092479887 +0000 UTC m=+75.862765402" watchObservedRunningTime="2026-07-23 17:06:20.09296305 +0000 UTC m=+75.863248572"
Jul 23 17:06:20.110469 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:20.110413    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-5945b5cfd-c8579" podStartSLOduration=2.268779952 podStartE2EDuration="7.110394964s" totalImagesPullingTime="4.841615012s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 17:06:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 17:06:20.109647855 +0000 UTC m=+75.879933399" watchObservedRunningTime="2026-07-23 17:06:20.110394964 +0000 UTC m=+75.880680481"
Jul 23 17:06:20.124885 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:20.124834    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-5c6b5" podStartSLOduration=68.181469736 podStartE2EDuration="1m15.124821432s" totalImagesPullingTime="6.943351696s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 17:05:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 17:06:20.124078398 +0000 UTC m=+75.894363913" watchObservedRunningTime="2026-07-23 17:06:20.124821432 +0000 UTC m=+75.895106948"
Jul 23 17:06:21.029947 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:21.029912    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:06:21.132107 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:21.132072    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:06:21.592209 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:21.592152    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:06:21.766601 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:21.766560    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:06:21.908391 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:21.908317    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:06:21.961418 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:21.961385    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-79f6958cdf-c2jxp"]
Jul 23 17:06:21.970782 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:21.970750    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-c2jxp"
Jul 23 17:06:21.974420 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:21.974396    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-7qzvq"]
Jul 23 17:06:21.974563 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:21.974437    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-f525r\""
Jul 23 17:06:21.975378 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:21.975313    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Jul 23 17:06:21.975524 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:21.975313    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Jul 23 17:06:21.976630 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:21.976611    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-7qzvq"
Jul 23 17:06:21.979422 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:21.979303    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-hvq9k\""
Jul 23 17:06:21.979570 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:21.979519    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 23 17:06:21.979721 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:21.979702    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 23 17:06:21.979961 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:21.979945    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 23 17:06:21.986808 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:21.986639    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-79f6958cdf-c2jxp"]
Jul 23 17:06:22.035420 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:22.035380    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:06:22.082630 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:22.082592    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8bc7\" (UniqueName: \"kubernetes.io/projected/c3cfd256-a951-47aa-a1ff-2604c290b07d-kube-api-access-t8bc7\") pod \"openshift-state-metrics-79f6958cdf-c2jxp\" (UID: \"c3cfd256-a951-47aa-a1ff-2604c290b07d\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-c2jxp"
Jul 23 17:06:22.082817 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:22.082646    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6mrd\" (UniqueName: \"kubernetes.io/projected/48493013-e9c4-4bb6-a9e2-060f45ad2f9c-kube-api-access-n6mrd\") pod \"node-exporter-7qzvq\" (UID: \"48493013-e9c4-4bb6-a9e2-060f45ad2f9c\") " pod="openshift-monitoring/node-exporter-7qzvq"
Jul 23 17:06:22.082817 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:22.082673    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/48493013-e9c4-4bb6-a9e2-060f45ad2f9c-metrics-client-ca\") pod \"node-exporter-7qzvq\" (UID: \"48493013-e9c4-4bb6-a9e2-060f45ad2f9c\") " pod="openshift-monitoring/node-exporter-7qzvq"
Jul 23 17:06:22.082817 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:22.082700    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/48493013-e9c4-4bb6-a9e2-060f45ad2f9c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-7qzvq\" (UID: \"48493013-e9c4-4bb6-a9e2-060f45ad2f9c\") " pod="openshift-monitoring/node-exporter-7qzvq"
Jul 23 17:06:22.082817 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:22.082721    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/48493013-e9c4-4bb6-a9e2-060f45ad2f9c-node-exporter-tls\") pod \"node-exporter-7qzvq\" (UID: \"48493013-e9c4-4bb6-a9e2-060f45ad2f9c\") " pod="openshift-monitoring/node-exporter-7qzvq"
Jul 23 17:06:22.082817 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:22.082753    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/48493013-e9c4-4bb6-a9e2-060f45ad2f9c-node-exporter-wtmp\") pod \"node-exporter-7qzvq\" (UID: \"48493013-e9c4-4bb6-a9e2-060f45ad2f9c\") " pod="openshift-monitoring/node-exporter-7qzvq"
Jul 23 17:06:22.082817 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:22.082804    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c3cfd256-a951-47aa-a1ff-2604c290b07d-metrics-client-ca\") pod \"openshift-state-metrics-79f6958cdf-c2jxp\" (UID: \"c3cfd256-a951-47aa-a1ff-2604c290b07d\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-c2jxp"
Jul 23 17:06:22.083083 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:22.082839    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/48493013-e9c4-4bb6-a9e2-060f45ad2f9c-sys\") pod \"node-exporter-7qzvq\" (UID: \"48493013-e9c4-4bb6-a9e2-060f45ad2f9c\") " pod="openshift-monitoring/node-exporter-7qzvq"
Jul 23 17:06:22.083083 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:22.082869    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/c3cfd256-a951-47aa-a1ff-2604c290b07d-openshift-state-metrics-tls\") pod \"openshift-state-metrics-79f6958cdf-c2jxp\" (UID: \"c3cfd256-a951-47aa-a1ff-2604c290b07d\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-c2jxp"
Jul 23 17:06:22.083083 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:22.082895    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/48493013-e9c4-4bb6-a9e2-060f45ad2f9c-node-exporter-textfile\") pod \"node-exporter-7qzvq\" (UID: \"48493013-e9c4-4bb6-a9e2-060f45ad2f9c\") " pod="openshift-monitoring/node-exporter-7qzvq"
Jul 23 17:06:22.083083 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:22.082916    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c3cfd256-a951-47aa-a1ff-2604c290b07d-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-79f6958cdf-c2jxp\" (UID: \"c3cfd256-a951-47aa-a1ff-2604c290b07d\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-c2jxp"
Jul 23 17:06:22.083083 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:22.082985    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/48493013-e9c4-4bb6-a9e2-060f45ad2f9c-node-exporter-accelerators-collector-config\") pod \"node-exporter-7qzvq\" (UID: \"48493013-e9c4-4bb6-a9e2-060f45ad2f9c\") " pod="openshift-monitoring/node-exporter-7qzvq"
Jul 23 17:06:22.083083 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:22.083004    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/48493013-e9c4-4bb6-a9e2-060f45ad2f9c-root\") pod \"node-exporter-7qzvq\" (UID: \"48493013-e9c4-4bb6-a9e2-060f45ad2f9c\") " pod="openshift-monitoring/node-exporter-7qzvq"
Jul 23 17:06:22.168032 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:22.167955    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:06:22.183949 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:22.183917    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6mrd\" (UniqueName: \"kubernetes.io/projected/48493013-e9c4-4bb6-a9e2-060f45ad2f9c-kube-api-access-n6mrd\") pod \"node-exporter-7qzvq\" (UID: \"48493013-e9c4-4bb6-a9e2-060f45ad2f9c\") " pod="openshift-monitoring/node-exporter-7qzvq"
Jul 23 17:06:22.184111 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:22.183955    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/48493013-e9c4-4bb6-a9e2-060f45ad2f9c-metrics-client-ca\") pod \"node-exporter-7qzvq\" (UID: \"48493013-e9c4-4bb6-a9e2-060f45ad2f9c\") " pod="openshift-monitoring/node-exporter-7qzvq"
Jul 23 17:06:22.184111 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:22.183988    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/48493013-e9c4-4bb6-a9e2-060f45ad2f9c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-7qzvq\" (UID: \"48493013-e9c4-4bb6-a9e2-060f45ad2f9c\") " pod="openshift-monitoring/node-exporter-7qzvq"
Jul 23 17:06:22.184111 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:22.184014    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/48493013-e9c4-4bb6-a9e2-060f45ad2f9c-node-exporter-tls\") pod \"node-exporter-7qzvq\" (UID: \"48493013-e9c4-4bb6-a9e2-060f45ad2f9c\") " pod="openshift-monitoring/node-exporter-7qzvq"
Jul 23 17:06:22.184111 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:22.184044    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/48493013-e9c4-4bb6-a9e2-060f45ad2f9c-node-exporter-wtmp\") pod \"node-exporter-7qzvq\" (UID: \"48493013-e9c4-4bb6-a9e2-060f45ad2f9c\") " pod="openshift-monitoring/node-exporter-7qzvq"
Jul 23 17:06:22.184111 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:22.184068    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c3cfd256-a951-47aa-a1ff-2604c290b07d-metrics-client-ca\") pod \"openshift-state-metrics-79f6958cdf-c2jxp\" (UID: \"c3cfd256-a951-47aa-a1ff-2604c290b07d\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-c2jxp"
Jul 23 17:06:22.184111 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:22.184090    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/48493013-e9c4-4bb6-a9e2-060f45ad2f9c-sys\") pod \"node-exporter-7qzvq\" (UID: \"48493013-e9c4-4bb6-a9e2-060f45ad2f9c\") " pod="openshift-monitoring/node-exporter-7qzvq"
Jul 23 17:06:22.184429 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:22.184121    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/c3cfd256-a951-47aa-a1ff-2604c290b07d-openshift-state-metrics-tls\") pod \"openshift-state-metrics-79f6958cdf-c2jxp\" (UID: \"c3cfd256-a951-47aa-a1ff-2604c290b07d\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-c2jxp"
Jul 23 17:06:22.184429 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:22.184154    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/48493013-e9c4-4bb6-a9e2-060f45ad2f9c-node-exporter-textfile\") pod \"node-exporter-7qzvq\" (UID: \"48493013-e9c4-4bb6-a9e2-060f45ad2f9c\") " pod="openshift-monitoring/node-exporter-7qzvq"
Jul 23 17:06:22.184429 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:22.184179    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c3cfd256-a951-47aa-a1ff-2604c290b07d-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-79f6958cdf-c2jxp\" (UID: \"c3cfd256-a951-47aa-a1ff-2604c290b07d\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-c2jxp"
Jul 23 17:06:22.184429 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:22.184223    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/48493013-e9c4-4bb6-a9e2-060f45ad2f9c-node-exporter-accelerators-collector-config\") pod \"node-exporter-7qzvq\" (UID: \"48493013-e9c4-4bb6-a9e2-060f45ad2f9c\") " pod="openshift-monitoring/node-exporter-7qzvq"
Jul 23 17:06:22.184429 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:22.184248    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/48493013-e9c4-4bb6-a9e2-060f45ad2f9c-node-exporter-wtmp\") pod \"node-exporter-7qzvq\" (UID: \"48493013-e9c4-4bb6-a9e2-060f45ad2f9c\") " pod="openshift-monitoring/node-exporter-7qzvq"
Jul 23 17:06:22.184429 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:22.184249    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/48493013-e9c4-4bb6-a9e2-060f45ad2f9c-root\") pod \"node-exporter-7qzvq\" (UID: \"48493013-e9c4-4bb6-a9e2-060f45ad2f9c\") " pod="openshift-monitoring/node-exporter-7qzvq"
Jul 23 17:06:22.184729 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:22.184628    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/48493013-e9c4-4bb6-a9e2-060f45ad2f9c-metrics-client-ca\") pod \"node-exporter-7qzvq\" (UID: \"48493013-e9c4-4bb6-a9e2-060f45ad2f9c\") " pod="openshift-monitoring/node-exporter-7qzvq"
Jul 23 17:06:22.184729 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:22.184700    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/48493013-e9c4-4bb6-a9e2-060f45ad2f9c-sys\") pod \"node-exporter-7qzvq\" (UID: \"48493013-e9c4-4bb6-a9e2-060f45ad2f9c\") " pod="openshift-monitoring/node-exporter-7qzvq"
Jul 23 17:06:22.185207 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:22.184860    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/48493013-e9c4-4bb6-a9e2-060f45ad2f9c-node-exporter-textfile\") pod \"node-exporter-7qzvq\" (UID: \"48493013-e9c4-4bb6-a9e2-060f45ad2f9c\") " pod="openshift-monitoring/node-exporter-7qzvq"
Jul 23 17:06:22.185207 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:22.185042    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c3cfd256-a951-47aa-a1ff-2604c290b07d-metrics-client-ca\") pod \"openshift-state-metrics-79f6958cdf-c2jxp\" (UID: \"c3cfd256-a951-47aa-a1ff-2604c290b07d\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-c2jxp"
Jul 23 17:06:22.185207 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:22.185055    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/48493013-e9c4-4bb6-a9e2-060f45ad2f9c-root\") pod \"node-exporter-7qzvq\" (UID: \"48493013-e9c4-4bb6-a9e2-060f45ad2f9c\") " pod="openshift-monitoring/node-exporter-7qzvq"
Jul 23 17:06:22.185207 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:22.185089    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8bc7\" (UniqueName: \"kubernetes.io/projected/c3cfd256-a951-47aa-a1ff-2604c290b07d-kube-api-access-t8bc7\") pod \"openshift-state-metrics-79f6958cdf-c2jxp\" (UID: \"c3cfd256-a951-47aa-a1ff-2604c290b07d\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-c2jxp"
Jul 23 17:06:22.185207 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:22.185169    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/48493013-e9c4-4bb6-a9e2-060f45ad2f9c-node-exporter-accelerators-collector-config\") pod \"node-exporter-7qzvq\" (UID: \"48493013-e9c4-4bb6-a9e2-060f45ad2f9c\") " pod="openshift-monitoring/node-exporter-7qzvq"
Jul 23 17:06:22.187272 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:22.187235    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/48493013-e9c4-4bb6-a9e2-060f45ad2f9c-node-exporter-tls\") pod \"node-exporter-7qzvq\" (UID: \"48493013-e9c4-4bb6-a9e2-060f45ad2f9c\") " pod="openshift-monitoring/node-exporter-7qzvq"
Jul 23 17:06:22.187434 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:22.187407    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/48493013-e9c4-4bb6-a9e2-060f45ad2f9c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-7qzvq\" (UID: \"48493013-e9c4-4bb6-a9e2-060f45ad2f9c\") " pod="openshift-monitoring/node-exporter-7qzvq"
Jul 23 17:06:22.187580 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:22.187563    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/c3cfd256-a951-47aa-a1ff-2604c290b07d-openshift-state-metrics-tls\") pod \"openshift-state-metrics-79f6958cdf-c2jxp\" (UID: \"c3cfd256-a951-47aa-a1ff-2604c290b07d\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-c2jxp"
Jul 23 17:06:22.187729 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:22.187707    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c3cfd256-a951-47aa-a1ff-2604c290b07d-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-79f6958cdf-c2jxp\" (UID: \"c3cfd256-a951-47aa-a1ff-2604c290b07d\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-c2jxp"
Jul 23 17:06:22.200355 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:22.200330    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8bc7\" (UniqueName: \"kubernetes.io/projected/c3cfd256-a951-47aa-a1ff-2604c290b07d-kube-api-access-t8bc7\") pod \"openshift-state-metrics-79f6958cdf-c2jxp\" (UID: \"c3cfd256-a951-47aa-a1ff-2604c290b07d\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-c2jxp"
Jul 23 17:06:22.201529 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:22.201501    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6mrd\" (UniqueName: \"kubernetes.io/projected/48493013-e9c4-4bb6-a9e2-060f45ad2f9c-kube-api-access-n6mrd\") pod \"node-exporter-7qzvq\" (UID: \"48493013-e9c4-4bb6-a9e2-060f45ad2f9c\") " pod="openshift-monitoring/node-exporter-7qzvq"
Jul 23 17:06:22.273913 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:22.273876    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:06:22.282856 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:22.282820    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-c2jxp"
Jul 23 17:06:22.290337 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:22.290308    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-7qzvq"
Jul 23 17:06:22.372637 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:22.372600    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:06:22.539993 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:22.539949    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:06:22.766853 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:06:22.766827    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48493013_e9c4_4bb6_a9e2_060f45ad2f9c.slice/crio-7db60f985fc2031083c27c57209c5661bcb6f5dcf7fb8357cd74ce063da4f9b5 WatchSource:0}: Error finding container 7db60f985fc2031083c27c57209c5661bcb6f5dcf7fb8357cd74ce063da4f9b5: Status 404 returned error can't find the container with id 7db60f985fc2031083c27c57209c5661bcb6f5dcf7fb8357cd74ce063da4f9b5
Jul 23 17:06:22.842477 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:22.842443    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:06:22.893765 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:22.893736    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-79f6958cdf-c2jxp"]
Jul 23 17:06:22.896416 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:06:22.896386    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3cfd256_a951_47aa_a1ff_2604c290b07d.slice/crio-cd167148c719a444b3707fd8ab77b1e779e335433ad7f8c318f9f84869059e89 WatchSource:0}: Error finding container cd167148c719a444b3707fd8ab77b1e779e335433ad7f8c318f9f84869059e89: Status 404 returned error can't find the container with id cd167148c719a444b3707fd8ab77b1e779e335433ad7f8c318f9f84869059e89
Jul 23 17:06:22.900873 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:22.900850    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 23 17:06:22.902973 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:22.902953    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 23 17:06:22.905996 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:22.905957    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Jul 23 17:06:22.906095 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:22.906003    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Jul 23 17:06:22.906095 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:22.906051    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Jul 23 17:06:22.906095 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:22.906079    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Jul 23 17:06:22.906294 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:22.905965    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Jul 23 17:06:22.906294 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:22.906009    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Jul 23 17:06:22.906294 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:22.906165    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Jul 23 17:06:22.906622 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:22.906602    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Jul 23 17:06:22.906962 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:22.906933    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-5kk55\""
Jul 23 17:06:22.906962 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:22.906948    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Jul 23 17:06:22.922959 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:22.922927    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 23 17:06:22.990243 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:22.990205    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/1f6e4496-99c7-4960-96d8-e1a50c52ac7c-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"1f6e4496-99c7-4960-96d8-e1a50c52ac7c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 17:06:22.990349 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:22.990292    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1f6e4496-99c7-4960-96d8-e1a50c52ac7c-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"1f6e4496-99c7-4960-96d8-e1a50c52ac7c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 17:06:22.990349 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:22.990322    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/1f6e4496-99c7-4960-96d8-e1a50c52ac7c-config-volume\") pod \"alertmanager-main-0\" (UID: \"1f6e4496-99c7-4960-96d8-e1a50c52ac7c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 17:06:22.990452 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:22.990350    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/1f6e4496-99c7-4960-96d8-e1a50c52ac7c-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"1f6e4496-99c7-4960-96d8-e1a50c52ac7c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 17:06:22.990452 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:22.990384    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/1f6e4496-99c7-4960-96d8-e1a50c52ac7c-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"1f6e4496-99c7-4960-96d8-e1a50c52ac7c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 17:06:22.990452 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:22.990414    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1f6e4496-99c7-4960-96d8-e1a50c52ac7c-tls-assets\") pod \"alertmanager-main-0\" (UID: \"1f6e4496-99c7-4960-96d8-e1a50c52ac7c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 17:06:22.990452 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:22.990436    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1f6e4496-99c7-4960-96d8-e1a50c52ac7c-web-config\") pod \"alertmanager-main-0\" (UID: \"1f6e4496-99c7-4960-96d8-e1a50c52ac7c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 17:06:22.990650 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:22.990518    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1f6e4496-99c7-4960-96d8-e1a50c52ac7c-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"1f6e4496-99c7-4960-96d8-e1a50c52ac7c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 17:06:22.990650 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:22.990572    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdvq6\" (UniqueName: \"kubernetes.io/projected/1f6e4496-99c7-4960-96d8-e1a50c52ac7c-kube-api-access-kdvq6\") pod \"alertmanager-main-0\" (UID: \"1f6e4496-99c7-4960-96d8-e1a50c52ac7c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 17:06:22.990650 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:22.990621    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1f6e4496-99c7-4960-96d8-e1a50c52ac7c-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"1f6e4496-99c7-4960-96d8-e1a50c52ac7c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 17:06:22.990767 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:22.990651    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1f6e4496-99c7-4960-96d8-e1a50c52ac7c-config-out\") pod \"alertmanager-main-0\" (UID: \"1f6e4496-99c7-4960-96d8-e1a50c52ac7c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 17:06:22.990767 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:22.990676    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1f6e4496-99c7-4960-96d8-e1a50c52ac7c-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"1f6e4496-99c7-4960-96d8-e1a50c52ac7c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 17:06:22.990767 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:22.990713    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/1f6e4496-99c7-4960-96d8-e1a50c52ac7c-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"1f6e4496-99c7-4960-96d8-e1a50c52ac7c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 17:06:23.045237 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:23.045158    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:06:23.091060 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:23.091034    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1f6e4496-99c7-4960-96d8-e1a50c52ac7c-config-out\") pod \"alertmanager-main-0\" (UID: \"1f6e4496-99c7-4960-96d8-e1a50c52ac7c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 17:06:23.091187 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:23.091064    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1f6e4496-99c7-4960-96d8-e1a50c52ac7c-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"1f6e4496-99c7-4960-96d8-e1a50c52ac7c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 17:06:23.091187 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:23.091090    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/1f6e4496-99c7-4960-96d8-e1a50c52ac7c-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"1f6e4496-99c7-4960-96d8-e1a50c52ac7c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 17:06:23.091187 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:23.091121    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/1f6e4496-99c7-4960-96d8-e1a50c52ac7c-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"1f6e4496-99c7-4960-96d8-e1a50c52ac7c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 17:06:23.091187 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:23.091151    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1f6e4496-99c7-4960-96d8-e1a50c52ac7c-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"1f6e4496-99c7-4960-96d8-e1a50c52ac7c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 17:06:23.091187 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:23.091175    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/1f6e4496-99c7-4960-96d8-e1a50c52ac7c-config-volume\") pod \"alertmanager-main-0\" (UID: \"1f6e4496-99c7-4960-96d8-e1a50c52ac7c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 17:06:23.091487 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:23.091201    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/1f6e4496-99c7-4960-96d8-e1a50c52ac7c-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"1f6e4496-99c7-4960-96d8-e1a50c52ac7c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 17:06:23.091487 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:23.091229    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/1f6e4496-99c7-4960-96d8-e1a50c52ac7c-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"1f6e4496-99c7-4960-96d8-e1a50c52ac7c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 17:06:23.091487 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:06:23.091248    2577 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found
Jul 23 17:06:23.091487 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:23.091275    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1f6e4496-99c7-4960-96d8-e1a50c52ac7c-tls-assets\") pod \"alertmanager-main-0\" (UID: \"1f6e4496-99c7-4960-96d8-e1a50c52ac7c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 17:06:23.091487 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:06:23.091338    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1f6e4496-99c7-4960-96d8-e1a50c52ac7c-secret-alertmanager-main-tls podName:1f6e4496-99c7-4960-96d8-e1a50c52ac7c nodeName:}" failed. No retries permitted until 2026-07-23 17:06:23.591317832 +0000 UTC m=+79.361603344 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/1f6e4496-99c7-4960-96d8-e1a50c52ac7c-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "1f6e4496-99c7-4960-96d8-e1a50c52ac7c") : secret "alertmanager-main-tls" not found
Jul 23 17:06:23.091487 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:23.091375    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1f6e4496-99c7-4960-96d8-e1a50c52ac7c-web-config\") pod \"alertmanager-main-0\" (UID: \"1f6e4496-99c7-4960-96d8-e1a50c52ac7c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 17:06:23.091487 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:23.091416    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1f6e4496-99c7-4960-96d8-e1a50c52ac7c-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"1f6e4496-99c7-4960-96d8-e1a50c52ac7c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 17:06:23.091487 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:23.091453    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdvq6\" (UniqueName: \"kubernetes.io/projected/1f6e4496-99c7-4960-96d8-e1a50c52ac7c-kube-api-access-kdvq6\") pod \"alertmanager-main-0\" (UID: \"1f6e4496-99c7-4960-96d8-e1a50c52ac7c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 17:06:23.091487 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:23.091487    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1f6e4496-99c7-4960-96d8-e1a50c52ac7c-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"1f6e4496-99c7-4960-96d8-e1a50c52ac7c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 17:06:23.091923 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:23.091596    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/1f6e4496-99c7-4960-96d8-e1a50c52ac7c-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"1f6e4496-99c7-4960-96d8-e1a50c52ac7c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 17:06:23.092310 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:23.092286    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1f6e4496-99c7-4960-96d8-e1a50c52ac7c-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"1f6e4496-99c7-4960-96d8-e1a50c52ac7c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 17:06:23.092480 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:23.092430    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1f6e4496-99c7-4960-96d8-e1a50c52ac7c-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"1f6e4496-99c7-4960-96d8-e1a50c52ac7c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 17:06:23.094309 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:23.094238    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1f6e4496-99c7-4960-96d8-e1a50c52ac7c-config-out\") pod \"alertmanager-main-0\" (UID: \"1f6e4496-99c7-4960-96d8-e1a50c52ac7c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 17:06:23.094603 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:23.094559    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/1f6e4496-99c7-4960-96d8-e1a50c52ac7c-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"1f6e4496-99c7-4960-96d8-e1a50c52ac7c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 17:06:23.094603 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:23.094565    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1f6e4496-99c7-4960-96d8-e1a50c52ac7c-tls-assets\") pod \"alertmanager-main-0\" (UID: \"1f6e4496-99c7-4960-96d8-e1a50c52ac7c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 17:06:23.094744 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:23.094620    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/1f6e4496-99c7-4960-96d8-e1a50c52ac7c-config-volume\") pod \"alertmanager-main-0\" (UID: \"1f6e4496-99c7-4960-96d8-e1a50c52ac7c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 17:06:23.095042 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:23.095010    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/1f6e4496-99c7-4960-96d8-e1a50c52ac7c-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"1f6e4496-99c7-4960-96d8-e1a50c52ac7c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 17:06:23.095159 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:23.095135    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1f6e4496-99c7-4960-96d8-e1a50c52ac7c-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"1f6e4496-99c7-4960-96d8-e1a50c52ac7c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 17:06:23.095317 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:23.095297    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1f6e4496-99c7-4960-96d8-e1a50c52ac7c-web-config\") pod \"alertmanager-main-0\" (UID: \"1f6e4496-99c7-4960-96d8-e1a50c52ac7c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 17:06:23.095839 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:23.095818    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1f6e4496-99c7-4960-96d8-e1a50c52ac7c-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"1f6e4496-99c7-4960-96d8-e1a50c52ac7c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 17:06:23.096536 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:23.096514    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7qzvq" event={"ID":"48493013-e9c4-4bb6-a9e2-060f45ad2f9c","Type":"ContainerStarted","Data":"7db60f985fc2031083c27c57209c5661bcb6f5dcf7fb8357cd74ce063da4f9b5"}
Jul 23 17:06:23.098485 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:23.098462    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-89shm" event={"ID":"0372b049-248a-43e5-84a8-198ec9bbc1cd","Type":"ContainerStarted","Data":"0f081d81935d22ff34a27ae6e6b457e7d2a3658a9ee3d2e4a93a1f149e701d2c"}
Jul 23 17:06:23.100121 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:23.100104    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-c2jxp" event={"ID":"c3cfd256-a951-47aa-a1ff-2604c290b07d","Type":"ContainerStarted","Data":"5c0140aee06c88c543b65f17bf98d071bc4cac34d3a5371e9c1b3494047f0b21"}
Jul 23 17:06:23.100205 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:23.100142    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-c2jxp" event={"ID":"c3cfd256-a951-47aa-a1ff-2604c290b07d","Type":"ContainerStarted","Data":"fb5798d9e1593d989ad1d44be5eea2861e4bfe503d9c51e7cb1966b21efd9c33"}
Jul 23 17:06:23.100205 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:23.100160    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-c2jxp" event={"ID":"c3cfd256-a951-47aa-a1ff-2604c290b07d","Type":"ContainerStarted","Data":"cd167148c719a444b3707fd8ab77b1e779e335433ad7f8c318f9f84869059e89"}
Jul 23 17:06:23.104347 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:23.104329    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdvq6\" (UniqueName: \"kubernetes.io/projected/1f6e4496-99c7-4960-96d8-e1a50c52ac7c-kube-api-access-kdvq6\") pod \"alertmanager-main-0\" (UID: \"1f6e4496-99c7-4960-96d8-e1a50c52ac7c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 17:06:23.122167 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:23.121085    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-89shm" podStartSLOduration=1.765681082 podStartE2EDuration="18.12106555s" totalImagesPullingTime="16.355384468s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 17:06:05 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 17:06:23.120205152 +0000 UTC m=+78.890490697" watchObservedRunningTime="2026-07-23 17:06:23.12106555 +0000 UTC m=+78.891351070"
Jul 23 17:06:23.147924 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:23.147898    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:06:23.252716 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:23.252670    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:06:23.485248 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:23.485211    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:06:23.585965 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:23.585927    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:06:23.595092 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:23.595068    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/1f6e4496-99c7-4960-96d8-e1a50c52ac7c-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"1f6e4496-99c7-4960-96d8-e1a50c52ac7c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 17:06:23.597736 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:23.597711    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/1f6e4496-99c7-4960-96d8-e1a50c52ac7c-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"1f6e4496-99c7-4960-96d8-e1a50c52ac7c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 23 17:06:23.812982 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:23.812953    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 23 17:06:23.955551 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:23.955526    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 23 17:06:23.957565 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:06:23.957539    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f6e4496_99c7_4960_96d8_e1a50c52ac7c.slice/crio-47259eef88a614a6133243dcf7f9fab08f28df6dbf21b1283a90f6edf7d70388 WatchSource:0}: Error finding container 47259eef88a614a6133243dcf7f9fab08f28df6dbf21b1283a90f6edf7d70388: Status 404 returned error can't find the container with id 47259eef88a614a6133243dcf7f9fab08f28df6dbf21b1283a90f6edf7d70388
Jul 23 17:06:24.062192 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:24.062127    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-9pw6q"
Jul 23 17:06:24.106318 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:24.106271    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1f6e4496-99c7-4960-96d8-e1a50c52ac7c","Type":"ContainerStarted","Data":"47259eef88a614a6133243dcf7f9fab08f28df6dbf21b1283a90f6edf7d70388"}
Jul 23 17:06:24.108265 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:24.108220    2577 generic.go:356] "Generic (PLEG): container finished" podID="48493013-e9c4-4bb6-a9e2-060f45ad2f9c" containerID="288a38541e5f5108ad3adbe07c4e01dfd45d20c480f2784a2270da578fcc57c7" exitCode=0
Jul 23 17:06:24.108387 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:24.108307    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7qzvq" event={"ID":"48493013-e9c4-4bb6-a9e2-060f45ad2f9c","Type":"ContainerDied","Data":"288a38541e5f5108ad3adbe07c4e01dfd45d20c480f2784a2270da578fcc57c7"}
Jul 23 17:06:24.849437 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:24.849403    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:06:24.929360 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:24.929328    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-5ff84d65c6-d2g5h"]
Jul 23 17:06:24.929721 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:24.929696    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:06:24.931955 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:24.931940    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-5ff84d65c6-d2g5h"
Jul 23 17:06:24.934651 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:24.934625    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-g8v8x\""
Jul 23 17:06:24.934917 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:24.934894    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Jul 23 17:06:24.935094 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:24.935080    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-91bf4d30qtp3\""
Jul 23 17:06:24.935352 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:24.935338    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Jul 23 17:06:24.935430 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:24.935366    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Jul 23 17:06:24.935641 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:24.935622    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Jul 23 17:06:24.935710 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:24.935654    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Jul 23 17:06:24.946575 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:24.946557    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-5ff84d65c6-d2g5h"]
Jul 23 17:06:25.005003 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:25.004969    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f50cede5-05ed-439c-b3f1-f2fa3c364d79-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5ff84d65c6-d2g5h\" (UID: \"f50cede5-05ed-439c-b3f1-f2fa3c364d79\") " pod="openshift-monitoring/thanos-querier-5ff84d65c6-d2g5h"
Jul 23 17:06:25.005155 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:25.005079    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f50cede5-05ed-439c-b3f1-f2fa3c364d79-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5ff84d65c6-d2g5h\" (UID: \"f50cede5-05ed-439c-b3f1-f2fa3c364d79\") " pod="openshift-monitoring/thanos-querier-5ff84d65c6-d2g5h"
Jul 23 17:06:25.005155 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:25.005125    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/f50cede5-05ed-439c-b3f1-f2fa3c364d79-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5ff84d65c6-d2g5h\" (UID: \"f50cede5-05ed-439c-b3f1-f2fa3c364d79\") " pod="openshift-monitoring/thanos-querier-5ff84d65c6-d2g5h"
Jul 23 17:06:25.005419 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:25.005153    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-527gb\" (UniqueName: \"kubernetes.io/projected/f50cede5-05ed-439c-b3f1-f2fa3c364d79-kube-api-access-527gb\") pod \"thanos-querier-5ff84d65c6-d2g5h\" (UID: \"f50cede5-05ed-439c-b3f1-f2fa3c364d79\") " pod="openshift-monitoring/thanos-querier-5ff84d65c6-d2g5h"
Jul 23 17:06:25.005419 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:25.005197    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/f50cede5-05ed-439c-b3f1-f2fa3c364d79-secret-thanos-querier-tls\") pod \"thanos-querier-5ff84d65c6-d2g5h\" (UID: \"f50cede5-05ed-439c-b3f1-f2fa3c364d79\") " pod="openshift-monitoring/thanos-querier-5ff84d65c6-d2g5h"
Jul 23 17:06:25.005419 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:25.005291    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f50cede5-05ed-439c-b3f1-f2fa3c364d79-metrics-client-ca\") pod \"thanos-querier-5ff84d65c6-d2g5h\" (UID: \"f50cede5-05ed-439c-b3f1-f2fa3c364d79\") " pod="openshift-monitoring/thanos-querier-5ff84d65c6-d2g5h"
Jul 23 17:06:25.005419 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:25.005364    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f50cede5-05ed-439c-b3f1-f2fa3c364d79-secret-grpc-tls\") pod \"thanos-querier-5ff84d65c6-d2g5h\" (UID: \"f50cede5-05ed-439c-b3f1-f2fa3c364d79\") " pod="openshift-monitoring/thanos-querier-5ff84d65c6-d2g5h"
Jul 23 17:06:25.005419 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:25.005392    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/f50cede5-05ed-439c-b3f1-f2fa3c364d79-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5ff84d65c6-d2g5h\" (UID: \"f50cede5-05ed-439c-b3f1-f2fa3c364d79\") " pod="openshift-monitoring/thanos-querier-5ff84d65c6-d2g5h"
Jul 23 17:06:25.106807 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:25.106731    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/f50cede5-05ed-439c-b3f1-f2fa3c364d79-secret-thanos-querier-tls\") pod \"thanos-querier-5ff84d65c6-d2g5h\" (UID: \"f50cede5-05ed-439c-b3f1-f2fa3c364d79\") " pod="openshift-monitoring/thanos-querier-5ff84d65c6-d2g5h"
Jul 23 17:06:25.106807 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:25.106764    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f50cede5-05ed-439c-b3f1-f2fa3c364d79-metrics-client-ca\") pod \"thanos-querier-5ff84d65c6-d2g5h\" (UID: \"f50cede5-05ed-439c-b3f1-f2fa3c364d79\") " pod="openshift-monitoring/thanos-querier-5ff84d65c6-d2g5h"
Jul 23 17:06:25.107234 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:25.106896    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f50cede5-05ed-439c-b3f1-f2fa3c364d79-secret-grpc-tls\") pod \"thanos-querier-5ff84d65c6-d2g5h\" (UID: \"f50cede5-05ed-439c-b3f1-f2fa3c364d79\") " pod="openshift-monitoring/thanos-querier-5ff84d65c6-d2g5h"
Jul 23 17:06:25.107234 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:25.106930    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/f50cede5-05ed-439c-b3f1-f2fa3c364d79-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5ff84d65c6-d2g5h\" (UID: \"f50cede5-05ed-439c-b3f1-f2fa3c364d79\") " pod="openshift-monitoring/thanos-querier-5ff84d65c6-d2g5h"
Jul 23 17:06:25.107234 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:25.106976    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f50cede5-05ed-439c-b3f1-f2fa3c364d79-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5ff84d65c6-d2g5h\" (UID: \"f50cede5-05ed-439c-b3f1-f2fa3c364d79\") " pod="openshift-monitoring/thanos-querier-5ff84d65c6-d2g5h"
Jul 23 17:06:25.107234 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:25.107043    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f50cede5-05ed-439c-b3f1-f2fa3c364d79-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5ff84d65c6-d2g5h\" (UID: \"f50cede5-05ed-439c-b3f1-f2fa3c364d79\") " pod="openshift-monitoring/thanos-querier-5ff84d65c6-d2g5h"
Jul 23 17:06:25.107234 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:25.107156    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/f50cede5-05ed-439c-b3f1-f2fa3c364d79-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5ff84d65c6-d2g5h\" (UID: \"f50cede5-05ed-439c-b3f1-f2fa3c364d79\") " pod="openshift-monitoring/thanos-querier-5ff84d65c6-d2g5h"
Jul 23 17:06:25.107234 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:25.107201    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-527gb\" (UniqueName: \"kubernetes.io/projected/f50cede5-05ed-439c-b3f1-f2fa3c364d79-kube-api-access-527gb\") pod \"thanos-querier-5ff84d65c6-d2g5h\" (UID: \"f50cede5-05ed-439c-b3f1-f2fa3c364d79\") " pod="openshift-monitoring/thanos-querier-5ff84d65c6-d2g5h"
Jul 23 17:06:25.107544 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:25.107524    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f50cede5-05ed-439c-b3f1-f2fa3c364d79-metrics-client-ca\") pod \"thanos-querier-5ff84d65c6-d2g5h\" (UID: \"f50cede5-05ed-439c-b3f1-f2fa3c364d79\") " pod="openshift-monitoring/thanos-querier-5ff84d65c6-d2g5h"
Jul 23 17:06:25.109819 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:25.109754    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/f50cede5-05ed-439c-b3f1-f2fa3c364d79-secret-thanos-querier-tls\") pod \"thanos-querier-5ff84d65c6-d2g5h\" (UID: \"f50cede5-05ed-439c-b3f1-f2fa3c364d79\") " pod="openshift-monitoring/thanos-querier-5ff84d65c6-d2g5h"
Jul 23 17:06:25.110003 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:25.109957    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f50cede5-05ed-439c-b3f1-f2fa3c364d79-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5ff84d65c6-d2g5h\" (UID: \"f50cede5-05ed-439c-b3f1-f2fa3c364d79\") " pod="openshift-monitoring/thanos-querier-5ff84d65c6-d2g5h"
Jul 23 17:06:25.110116 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:25.110014    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f50cede5-05ed-439c-b3f1-f2fa3c364d79-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5ff84d65c6-d2g5h\" (UID: \"f50cede5-05ed-439c-b3f1-f2fa3c364d79\") " pod="openshift-monitoring/thanos-querier-5ff84d65c6-d2g5h"
Jul 23 17:06:25.110116 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:25.110030    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f50cede5-05ed-439c-b3f1-f2fa3c364d79-secret-grpc-tls\") pod \"thanos-querier-5ff84d65c6-d2g5h\" (UID: \"f50cede5-05ed-439c-b3f1-f2fa3c364d79\") " pod="openshift-monitoring/thanos-querier-5ff84d65c6-d2g5h"
Jul 23 17:06:25.110365 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:25.110341    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/f50cede5-05ed-439c-b3f1-f2fa3c364d79-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5ff84d65c6-d2g5h\" (UID: \"f50cede5-05ed-439c-b3f1-f2fa3c364d79\") " pod="openshift-monitoring/thanos-querier-5ff84d65c6-d2g5h"
Jul 23 17:06:25.110942 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:25.110921    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/f50cede5-05ed-439c-b3f1-f2fa3c364d79-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5ff84d65c6-d2g5h\" (UID: \"f50cede5-05ed-439c-b3f1-f2fa3c364d79\") " pod="openshift-monitoring/thanos-querier-5ff84d65c6-d2g5h"
Jul 23 17:06:25.114449 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:25.114420    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7qzvq" event={"ID":"48493013-e9c4-4bb6-a9e2-060f45ad2f9c","Type":"ContainerStarted","Data":"35e8bb5e90be84a849cc177c545e1cd3daf9f95b9d190813878fbcc002586ca5"}
Jul 23 17:06:25.114548 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:25.114488    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7qzvq" event={"ID":"48493013-e9c4-4bb6-a9e2-060f45ad2f9c","Type":"ContainerStarted","Data":"ab9fe9c683d17aa6099cf0381c7b7415c06567174109136ccce7729d62a832bc"}
Jul 23 17:06:25.116195 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:25.116173    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-527gb\" (UniqueName: \"kubernetes.io/projected/f50cede5-05ed-439c-b3f1-f2fa3c364d79-kube-api-access-527gb\") pod \"thanos-querier-5ff84d65c6-d2g5h\" (UID: \"f50cede5-05ed-439c-b3f1-f2fa3c364d79\") " pod="openshift-monitoring/thanos-querier-5ff84d65c6-d2g5h"
Jul 23 17:06:25.116586 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:25.116565    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-c2jxp" event={"ID":"c3cfd256-a951-47aa-a1ff-2604c290b07d","Type":"ContainerStarted","Data":"c617ba12133a90c738452d10389c87246ed73b902f72d8d28ea399fc19d1d4c1"}
Jul 23 17:06:25.133483 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:25.133446    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-7qzvq" podStartSLOduration=3.218087033 podStartE2EDuration="4.133434515s" totalImagesPullingTime="850.309232ms" totalInitContainerRuntime="65.03825ms" isStatefulPod=true podCreationTimestamp="2026-07-23 17:06:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 17:06:25.132872162 +0000 UTC m=+80.903157682" watchObservedRunningTime="2026-07-23 17:06:25.133434515 +0000 UTC m=+80.903720034"
Jul 23 17:06:25.153707 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:25.153662    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-c2jxp" podStartSLOduration=2.911150394 podStartE2EDuration="4.153649386s" totalImagesPullingTime="1.242498992s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 17:06:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 17:06:25.15279075 +0000 UTC m=+80.923076288" watchObservedRunningTime="2026-07-23 17:06:25.153649386 +0000 UTC m=+80.923934905"
Jul 23 17:06:25.241324 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:25.241294    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-5ff84d65c6-d2g5h"
Jul 23 17:06:25.276674 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:25.276633    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:06:25.369894 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:25.369820    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:06:25.389750 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:25.389726    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-5ff84d65c6-d2g5h"]
Jul 23 17:06:25.496373 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:25.496339    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:06:25.606338 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:06:25.606296    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf50cede5_05ed_439c_b3f1_f2fa3c364d79.slice/crio-ededfe282b9eb27449e244d3894f5a89cc654006047e48524ce035375246ed39 WatchSource:0}: Error finding container ededfe282b9eb27449e244d3894f5a89cc654006047e48524ce035375246ed39: Status 404 returned error can't find the container with id ededfe282b9eb27449e244d3894f5a89cc654006047e48524ce035375246ed39
Jul 23 17:06:25.717833 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:25.717807    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:06:26.122021 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:26.121992    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5ff84d65c6-d2g5h" event={"ID":"f50cede5-05ed-439c-b3f1-f2fa3c364d79","Type":"ContainerStarted","Data":"ededfe282b9eb27449e244d3894f5a89cc654006047e48524ce035375246ed39"}
Jul 23 17:06:26.123504 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:26.123481    2577 generic.go:356] "Generic (PLEG): container finished" podID="1f6e4496-99c7-4960-96d8-e1a50c52ac7c" containerID="d84b02a0884dbbc1be69c0030ad8974b9ea507ea73c445618edd7e558fac0c85" exitCode=0
Jul 23 17:06:26.123595 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:26.123555    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1f6e4496-99c7-4960-96d8-e1a50c52ac7c","Type":"ContainerDied","Data":"d84b02a0884dbbc1be69c0030ad8974b9ea507ea73c445618edd7e558fac0c85"}
Jul 23 17:06:26.378484 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:26.378409    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-69d5f9b95d-ptst2"]
Jul 23 17:06:26.380557 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:26.380542    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-69d5f9b95d-ptst2"
Jul 23 17:06:26.383177 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:26.383123    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Jul 23 17:06:26.383177 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:26.383154    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 23 17:06:26.383177 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:26.383154    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Jul 23 17:06:26.384211 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:26.384193    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-ec0v7hka2c02k\""
Jul 23 17:06:26.384329 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:26.384296    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-x7xrf\""
Jul 23 17:06:26.384400 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:26.384382    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Jul 23 17:06:26.394523 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:26.394503    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-69d5f9b95d-ptst2"]
Jul 23 17:06:26.521713 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:26.521598    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/33724e8d-1c37-43f7-bafe-01809d1b90ed-metrics-server-audit-profiles\") pod \"metrics-server-69d5f9b95d-ptst2\" (UID: \"33724e8d-1c37-43f7-bafe-01809d1b90ed\") " pod="openshift-monitoring/metrics-server-69d5f9b95d-ptst2"
Jul 23 17:06:26.521713 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:26.521647    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/33724e8d-1c37-43f7-bafe-01809d1b90ed-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-69d5f9b95d-ptst2\" (UID: \"33724e8d-1c37-43f7-bafe-01809d1b90ed\") " pod="openshift-monitoring/metrics-server-69d5f9b95d-ptst2"
Jul 23 17:06:26.521713 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:26.521697    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/33724e8d-1c37-43f7-bafe-01809d1b90ed-secret-metrics-server-client-certs\") pod \"metrics-server-69d5f9b95d-ptst2\" (UID: \"33724e8d-1c37-43f7-bafe-01809d1b90ed\") " pod="openshift-monitoring/metrics-server-69d5f9b95d-ptst2"
Jul 23 17:06:26.521975 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:26.521808    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/33724e8d-1c37-43f7-bafe-01809d1b90ed-secret-metrics-server-tls\") pod \"metrics-server-69d5f9b95d-ptst2\" (UID: \"33724e8d-1c37-43f7-bafe-01809d1b90ed\") " pod="openshift-monitoring/metrics-server-69d5f9b95d-ptst2"
Jul 23 17:06:26.521975 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:26.521844    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/33724e8d-1c37-43f7-bafe-01809d1b90ed-client-ca-bundle\") pod \"metrics-server-69d5f9b95d-ptst2\" (UID: \"33724e8d-1c37-43f7-bafe-01809d1b90ed\") " pod="openshift-monitoring/metrics-server-69d5f9b95d-ptst2"
Jul 23 17:06:26.521975 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:26.521869    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lgvtq\" (UniqueName: \"kubernetes.io/projected/33724e8d-1c37-43f7-bafe-01809d1b90ed-kube-api-access-lgvtq\") pod \"metrics-server-69d5f9b95d-ptst2\" (UID: \"33724e8d-1c37-43f7-bafe-01809d1b90ed\") " pod="openshift-monitoring/metrics-server-69d5f9b95d-ptst2"
Jul 23 17:06:26.521975 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:26.521937    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/33724e8d-1c37-43f7-bafe-01809d1b90ed-audit-log\") pod \"metrics-server-69d5f9b95d-ptst2\" (UID: \"33724e8d-1c37-43f7-bafe-01809d1b90ed\") " pod="openshift-monitoring/metrics-server-69d5f9b95d-ptst2"
Jul 23 17:06:26.619567 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:26.619538    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:06:26.622739 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:26.622710    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/33724e8d-1c37-43f7-bafe-01809d1b90ed-secret-metrics-server-client-certs\") pod \"metrics-server-69d5f9b95d-ptst2\" (UID: \"33724e8d-1c37-43f7-bafe-01809d1b90ed\") " pod="openshift-monitoring/metrics-server-69d5f9b95d-ptst2"
Jul 23 17:06:26.622950 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:26.622751    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/33724e8d-1c37-43f7-bafe-01809d1b90ed-secret-metrics-server-tls\") pod \"metrics-server-69d5f9b95d-ptst2\" (UID: \"33724e8d-1c37-43f7-bafe-01809d1b90ed\") " pod="openshift-monitoring/metrics-server-69d5f9b95d-ptst2"
Jul 23 17:06:26.622950 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:26.622770    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/33724e8d-1c37-43f7-bafe-01809d1b90ed-client-ca-bundle\") pod \"metrics-server-69d5f9b95d-ptst2\" (UID: \"33724e8d-1c37-43f7-bafe-01809d1b90ed\") " pod="openshift-monitoring/metrics-server-69d5f9b95d-ptst2"
Jul 23 17:06:26.622950 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:26.622792    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgvtq\" (UniqueName: \"kubernetes.io/projected/33724e8d-1c37-43f7-bafe-01809d1b90ed-kube-api-access-lgvtq\") pod \"metrics-server-69d5f9b95d-ptst2\" (UID: \"33724e8d-1c37-43f7-bafe-01809d1b90ed\") " pod="openshift-monitoring/metrics-server-69d5f9b95d-ptst2"
Jul 23 17:06:26.622950 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:26.622829    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/33724e8d-1c37-43f7-bafe-01809d1b90ed-audit-log\") pod \"metrics-server-69d5f9b95d-ptst2\" (UID: \"33724e8d-1c37-43f7-bafe-01809d1b90ed\") " pod="openshift-monitoring/metrics-server-69d5f9b95d-ptst2"
Jul 23 17:06:26.622950 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:26.622863    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/33724e8d-1c37-43f7-bafe-01809d1b90ed-metrics-server-audit-profiles\") pod \"metrics-server-69d5f9b95d-ptst2\" (UID: \"33724e8d-1c37-43f7-bafe-01809d1b90ed\") " pod="openshift-monitoring/metrics-server-69d5f9b95d-ptst2"
Jul 23 17:06:26.622950 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:26.622893    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/33724e8d-1c37-43f7-bafe-01809d1b90ed-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-69d5f9b95d-ptst2\" (UID: \"33724e8d-1c37-43f7-bafe-01809d1b90ed\") " pod="openshift-monitoring/metrics-server-69d5f9b95d-ptst2"
Jul 23 17:06:26.623414 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:26.623367    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/33724e8d-1c37-43f7-bafe-01809d1b90ed-audit-log\") pod \"metrics-server-69d5f9b95d-ptst2\" (UID: \"33724e8d-1c37-43f7-bafe-01809d1b90ed\") " pod="openshift-monitoring/metrics-server-69d5f9b95d-ptst2"
Jul 23 17:06:26.623915 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:26.623878    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/33724e8d-1c37-43f7-bafe-01809d1b90ed-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-69d5f9b95d-ptst2\" (UID: \"33724e8d-1c37-43f7-bafe-01809d1b90ed\") " pod="openshift-monitoring/metrics-server-69d5f9b95d-ptst2"
Jul 23 17:06:26.624289 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:26.624241    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/33724e8d-1c37-43f7-bafe-01809d1b90ed-metrics-server-audit-profiles\") pod \"metrics-server-69d5f9b95d-ptst2\" (UID: \"33724e8d-1c37-43f7-bafe-01809d1b90ed\") " pod="openshift-monitoring/metrics-server-69d5f9b95d-ptst2"
Jul 23 17:06:26.625600 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:26.625581    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/33724e8d-1c37-43f7-bafe-01809d1b90ed-client-ca-bundle\") pod \"metrics-server-69d5f9b95d-ptst2\" (UID: \"33724e8d-1c37-43f7-bafe-01809d1b90ed\") " pod="openshift-monitoring/metrics-server-69d5f9b95d-ptst2"
Jul 23 17:06:26.625853 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:26.625833    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/33724e8d-1c37-43f7-bafe-01809d1b90ed-secret-metrics-server-client-certs\") pod \"metrics-server-69d5f9b95d-ptst2\" (UID: \"33724e8d-1c37-43f7-bafe-01809d1b90ed\") " pod="openshift-monitoring/metrics-server-69d5f9b95d-ptst2"
Jul 23 17:06:26.625977 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:26.625960    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/33724e8d-1c37-43f7-bafe-01809d1b90ed-secret-metrics-server-tls\") pod \"metrics-server-69d5f9b95d-ptst2\" (UID: \"33724e8d-1c37-43f7-bafe-01809d1b90ed\") " pod="openshift-monitoring/metrics-server-69d5f9b95d-ptst2"
Jul 23 17:06:26.646297 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:26.646211    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lgvtq\" (UniqueName: \"kubernetes.io/projected/33724e8d-1c37-43f7-bafe-01809d1b90ed-kube-api-access-lgvtq\") pod \"metrics-server-69d5f9b95d-ptst2\" (UID: \"33724e8d-1c37-43f7-bafe-01809d1b90ed\") " pod="openshift-monitoring/metrics-server-69d5f9b95d-ptst2"
Jul 23 17:06:26.646466 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:26.646445    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-7b7b7bc8bf-b9gk6"]
Jul 23 17:06:26.649383 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:26.649353    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-b9gk6"
Jul 23 17:06:26.651766 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:26.651748    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Jul 23 17:06:26.651920 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:26.651895    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-8g8m4\""
Jul 23 17:06:26.660038 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:26.660014    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-7b7b7bc8bf-b9gk6"]
Jul 23 17:06:26.690124 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:26.690076    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-69d5f9b95d-ptst2"
Jul 23 17:06:26.722126 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:26.722091    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:06:26.724299 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:26.724275    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3530bb86-ef19-4c9c-a150-920b342ad3e7-monitoring-plugin-cert\") pod \"monitoring-plugin-7b7b7bc8bf-b9gk6\" (UID: \"3530bb86-ef19-4c9c-a150-920b342ad3e7\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-b9gk6"
Jul 23 17:06:26.724406 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:26.724330    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-txlzw\" (UniqueName: \"kubernetes.io/projected/3530bb86-ef19-4c9c-a150-920b342ad3e7-kube-api-access-txlzw\") pod \"monitoring-plugin-7b7b7bc8bf-b9gk6\" (UID: \"3530bb86-ef19-4c9c-a150-920b342ad3e7\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-b9gk6"
Jul 23 17:06:26.824803 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:26.824764    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3530bb86-ef19-4c9c-a150-920b342ad3e7-monitoring-plugin-cert\") pod \"monitoring-plugin-7b7b7bc8bf-b9gk6\" (UID: \"3530bb86-ef19-4c9c-a150-920b342ad3e7\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-b9gk6"
Jul 23 17:06:26.824972 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:26.824836    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txlzw\" (UniqueName: \"kubernetes.io/projected/3530bb86-ef19-4c9c-a150-920b342ad3e7-kube-api-access-txlzw\") pod \"monitoring-plugin-7b7b7bc8bf-b9gk6\" (UID: \"3530bb86-ef19-4c9c-a150-920b342ad3e7\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-b9gk6"
Jul 23 17:06:26.828239 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:26.828208    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3530bb86-ef19-4c9c-a150-920b342ad3e7-monitoring-plugin-cert\") pod \"monitoring-plugin-7b7b7bc8bf-b9gk6\" (UID: \"3530bb86-ef19-4c9c-a150-920b342ad3e7\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-b9gk6"
Jul 23 17:06:26.829745 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:26.829706    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:06:26.834834 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:26.834791    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-txlzw\" (UniqueName: \"kubernetes.io/projected/3530bb86-ef19-4c9c-a150-920b342ad3e7-kube-api-access-txlzw\") pod \"monitoring-plugin-7b7b7bc8bf-b9gk6\" (UID: \"3530bb86-ef19-4c9c-a150-920b342ad3e7\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-b9gk6"
Jul 23 17:06:26.838704 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:26.838683    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-69d5f9b95d-ptst2"]
Jul 23 17:06:26.842490 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:06:26.842450    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33724e8d_1c37_43f7_bafe_01809d1b90ed.slice/crio-c7026266532a581b18c4889ad239433ae8e898fbb56f0686edd2d52c886c4856 WatchSource:0}: Error finding container c7026266532a581b18c4889ad239433ae8e898fbb56f0686edd2d52c886c4856: Status 404 returned error can't find the container with id c7026266532a581b18c4889ad239433ae8e898fbb56f0686edd2d52c886c4856
Jul 23 17:06:26.922789 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:26.922715    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:06:26.964013 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:26.963959    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-b9gk6"
Jul 23 17:06:27.126655 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:27.126629    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-7b7b7bc8bf-b9gk6"]
Jul 23 17:06:27.128950 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:06:27.128918    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3530bb86_ef19_4c9c_a150_920b342ad3e7.slice/crio-534ffafcc81d5c3d8e44e0f6704cfdeed416d0454c0ec22ace190c373c045c0a WatchSource:0}: Error finding container 534ffafcc81d5c3d8e44e0f6704cfdeed416d0454c0ec22ace190c373c045c0a: Status 404 returned error can't find the container with id 534ffafcc81d5c3d8e44e0f6704cfdeed416d0454c0ec22ace190c373c045c0a
Jul 23 17:06:27.129499 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:27.129475    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-69d5f9b95d-ptst2" event={"ID":"33724e8d-1c37-43f7-bafe-01809d1b90ed","Type":"ContainerStarted","Data":"c7026266532a581b18c4889ad239433ae8e898fbb56f0686edd2d52c886c4856"}
Jul 23 17:06:28.135544 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:28.135496    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-b9gk6" event={"ID":"3530bb86-ef19-4c9c-a150-920b342ad3e7","Type":"ContainerStarted","Data":"534ffafcc81d5c3d8e44e0f6704cfdeed416d0454c0ec22ace190c373c045c0a"}
Jul 23 17:06:28.275802 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:28.275728    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:06:28.366020 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:28.365976    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:06:30.144544 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:30.144505    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5ff84d65c6-d2g5h" event={"ID":"f50cede5-05ed-439c-b3f1-f2fa3c364d79","Type":"ContainerStarted","Data":"ca3a15763bc6f2066b6e8b8f4c858e5dd1b957ff0473c0883766fe6c22722551"}
Jul 23 17:06:30.144978 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:30.144606    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5ff84d65c6-d2g5h" event={"ID":"f50cede5-05ed-439c-b3f1-f2fa3c364d79","Type":"ContainerStarted","Data":"952f1c95fa1263120bb51f0fe693f9db926122de91b87b01c616c42f82b74bba"}
Jul 23 17:06:30.144978 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:30.144647    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5ff84d65c6-d2g5h" event={"ID":"f50cede5-05ed-439c-b3f1-f2fa3c364d79","Type":"ContainerStarted","Data":"bee0fc9d4365c6420341ee8f6f57bc36baef272f559c347f67872b0378d976d7"}
Jul 23 17:06:30.146026 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:30.145997    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-69d5f9b95d-ptst2" event={"ID":"33724e8d-1c37-43f7-bafe-01809d1b90ed","Type":"ContainerStarted","Data":"a290f577237d62a2cb533030d2e23ac9899cc9e93028d848165943354a014801"}
Jul 23 17:06:30.149119 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:30.149091    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1f6e4496-99c7-4960-96d8-e1a50c52ac7c","Type":"ContainerStarted","Data":"00da2f92d70aba513292f7e301fc333c7375891c6667e94bc578d7f58a2cb834"}
Jul 23 17:06:30.149236 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:30.149162    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1f6e4496-99c7-4960-96d8-e1a50c52ac7c","Type":"ContainerStarted","Data":"68f261cc291d4b355ccaf2ffee5ea20c11032b3433bd916de51a42cbf65c8a50"}
Jul 23 17:06:30.149236 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:30.149206    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1f6e4496-99c7-4960-96d8-e1a50c52ac7c","Type":"ContainerStarted","Data":"dbe86dae521d6cf779d1782c05bffc7cbf9fb313854fd4b9a68515b03a3de4df"}
Jul 23 17:06:30.149374 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:30.149246    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1f6e4496-99c7-4960-96d8-e1a50c52ac7c","Type":"ContainerStarted","Data":"439ca7b632678388d326ff150a652f3223867b0a6ffd4e5044077ea7ac937f4c"}
Jul 23 17:06:30.149374 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:30.149290    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1f6e4496-99c7-4960-96d8-e1a50c52ac7c","Type":"ContainerStarted","Data":"75de4b895454a8ed59e0aa9c680997acdacec7f389067e3a4c83070f27a48e93"}
Jul 23 17:06:30.150671 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:30.150651    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-b9gk6" event={"ID":"3530bb86-ef19-4c9c-a150-920b342ad3e7","Type":"ContainerStarted","Data":"c206c9fc83fdf8127ba6b41d40ad3635a18a527fba83b798dc1cb9053773774b"}
Jul 23 17:06:30.150856 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:30.150842    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-b9gk6"
Jul 23 17:06:30.155626 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:30.155604    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-b9gk6"
Jul 23 17:06:30.168298 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:30.168234    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-69d5f9b95d-ptst2" podStartSLOduration=1.790954992 podStartE2EDuration="4.16821771s" totalImagesPullingTime="2.377262718s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 17:06:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 17:06:30.167574249 +0000 UTC m=+85.937859772" watchObservedRunningTime="2026-07-23 17:06:30.16821771 +0000 UTC m=+85.938503230"
Jul 23 17:06:30.193922 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:30.193878    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-b9gk6" podStartSLOduration=2.062569675 podStartE2EDuration="4.193859681s" totalImagesPullingTime="2.131290006s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 17:06:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 17:06:30.192629888 +0000 UTC m=+85.962915407" watchObservedRunningTime="2026-07-23 17:06:30.193859681 +0000 UTC m=+85.964145201"
Jul 23 17:06:30.381767 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:30.381735    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:06:30.481506 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:30.481457    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:06:31.157121 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:31.157026    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1f6e4496-99c7-4960-96d8-e1a50c52ac7c","Type":"ContainerStarted","Data":"2934486122c193a30ef8b6b33828f7d1509983f79cd16c59978623ead97a46f1"}
Jul 23 17:06:31.159826 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:31.159797    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5ff84d65c6-d2g5h" event={"ID":"f50cede5-05ed-439c-b3f1-f2fa3c364d79","Type":"ContainerStarted","Data":"482cb04bd7d5e8ef55d79857bf1008abed7e652e4385b07da4fd1d4b1c9eb740"}
Jul 23 17:06:31.159949 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:31.159858    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5ff84d65c6-d2g5h" event={"ID":"f50cede5-05ed-439c-b3f1-f2fa3c364d79","Type":"ContainerStarted","Data":"23c0b0a86635af5918692e61be69349fafe730a22a2e65f0bcd0d65cf2fdd58b"}
Jul 23 17:06:31.159949 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:31.159884    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5ff84d65c6-d2g5h" event={"ID":"f50cede5-05ed-439c-b3f1-f2fa3c364d79","Type":"ContainerStarted","Data":"ccd4b149bd1e4dc5124fbd84fb50eb7663c40cdb32d9e8be5891c0c906522d5d"}
Jul 23 17:06:31.192951 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:31.192896    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.411853165 podStartE2EDuration="9.192882386s" totalImagesPullingTime="5.765445007s" totalInitContainerRuntime="15.584214ms" isStatefulPod=false podCreationTimestamp="2026-07-23 17:06:22 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 17:06:31.191975738 +0000 UTC m=+86.962261257" watchObservedRunningTime="2026-07-23 17:06:31.192882386 +0000 UTC m=+86.963167903"
Jul 23 17:06:31.230966 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:31.230906    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-5ff84d65c6-d2g5h" podStartSLOduration=2.501229449 podStartE2EDuration="7.230883194s" totalImagesPullingTime="4.729653745s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 17:06:24 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 17:06:31.229365304 +0000 UTC m=+86.999650824" watchObservedRunningTime="2026-07-23 17:06:31.230883194 +0000 UTC m=+87.001168715"
Jul 23 17:06:32.164312 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:32.164283    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-5ff84d65c6-d2g5h"
Jul 23 17:06:32.283310 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:32.283276    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:06:32.382212 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:32.382181    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:06:37.421065 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:37.421036    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:06:37.525743 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:37.525711    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:06:37.627614 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:37.627577    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:06:38.175062 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:38.175029    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-5ff84d65c6-d2g5h"
Jul 23 17:06:40.533452 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:40.533397    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:06:40.608355 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:40.608321    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:06:41.271653 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:41.271621    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:06:46.690599 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:46.690553    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-69d5f9b95d-ptst2"
Jul 23 17:06:46.690973 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:46.690648    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-69d5f9b95d-ptst2"
Jul 23 17:06:51.088411 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:51.088376    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-5c6b5"
Jul 23 17:06:55.482159 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:55.482128    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:06:55.550485 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:06:55.550453    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:07:05.096610 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:07:05.096573    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:07:06.696990 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:07:06.696958    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-69d5f9b95d-ptst2"
Jul 23 17:07:06.700761 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:07:06.700732    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-69d5f9b95d-ptst2"
Jul 23 17:07:41.213190 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:07:41.213147    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:07:41.460242 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:07:41.460208    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:07:46.314362 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:07:46.314328    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-56745b4487-g2wzx"]
Jul 23 17:07:46.317596 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:07:46.317579    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-56745b4487-g2wzx"
Jul 23 17:07:46.320370 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:07:46.320345    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Jul 23 17:07:46.321063 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:07:46.320962    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Jul 23 17:07:46.321248 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:07:46.321084    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Jul 23 17:07:46.321248 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:07:46.321097    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Jul 23 17:07:46.321473 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:07:46.321455    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-mmrvs\""
Jul 23 17:07:46.321535 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:07:46.321491    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Jul 23 17:07:46.325036 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:07:46.325013    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Jul 23 17:07:46.335121 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:07:46.335097    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-56745b4487-g2wzx"]
Jul 23 17:07:46.424344 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:07:46.424312    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d128b7fd-71ee-4da9-8d06-bdcb68513773-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-56745b4487-g2wzx\" (UID: \"d128b7fd-71ee-4da9-8d06-bdcb68513773\") " pod="openshift-monitoring/telemeter-client-56745b4487-g2wzx"
Jul 23 17:07:46.424512 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:07:46.424353    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d128b7fd-71ee-4da9-8d06-bdcb68513773-telemeter-trusted-ca-bundle\") pod \"telemeter-client-56745b4487-g2wzx\" (UID: \"d128b7fd-71ee-4da9-8d06-bdcb68513773\") " pod="openshift-monitoring/telemeter-client-56745b4487-g2wzx"
Jul 23 17:07:46.424512 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:07:46.424419    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d128b7fd-71ee-4da9-8d06-bdcb68513773-serving-certs-ca-bundle\") pod \"telemeter-client-56745b4487-g2wzx\" (UID: \"d128b7fd-71ee-4da9-8d06-bdcb68513773\") " pod="openshift-monitoring/telemeter-client-56745b4487-g2wzx"
Jul 23 17:07:46.424512 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:07:46.424467    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/d128b7fd-71ee-4da9-8d06-bdcb68513773-federate-client-tls\") pod \"telemeter-client-56745b4487-g2wzx\" (UID: \"d128b7fd-71ee-4da9-8d06-bdcb68513773\") " pod="openshift-monitoring/telemeter-client-56745b4487-g2wzx"
Jul 23 17:07:46.424615 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:07:46.424517    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d128b7fd-71ee-4da9-8d06-bdcb68513773-metrics-client-ca\") pod \"telemeter-client-56745b4487-g2wzx\" (UID: \"d128b7fd-71ee-4da9-8d06-bdcb68513773\") " pod="openshift-monitoring/telemeter-client-56745b4487-g2wzx"
Jul 23 17:07:46.424615 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:07:46.424560    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/d128b7fd-71ee-4da9-8d06-bdcb68513773-secret-telemeter-client\") pod \"telemeter-client-56745b4487-g2wzx\" (UID: \"d128b7fd-71ee-4da9-8d06-bdcb68513773\") " pod="openshift-monitoring/telemeter-client-56745b4487-g2wzx"
Jul 23 17:07:46.424615 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:07:46.424597    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/d128b7fd-71ee-4da9-8d06-bdcb68513773-telemeter-client-tls\") pod \"telemeter-client-56745b4487-g2wzx\" (UID: \"d128b7fd-71ee-4da9-8d06-bdcb68513773\") " pod="openshift-monitoring/telemeter-client-56745b4487-g2wzx"
Jul 23 17:07:46.424704 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:07:46.424621    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5x9sw\" (UniqueName: \"kubernetes.io/projected/d128b7fd-71ee-4da9-8d06-bdcb68513773-kube-api-access-5x9sw\") pod \"telemeter-client-56745b4487-g2wzx\" (UID: \"d128b7fd-71ee-4da9-8d06-bdcb68513773\") " pod="openshift-monitoring/telemeter-client-56745b4487-g2wzx"
Jul 23 17:07:46.525899 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:07:46.525862    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/d128b7fd-71ee-4da9-8d06-bdcb68513773-federate-client-tls\") pod \"telemeter-client-56745b4487-g2wzx\" (UID: \"d128b7fd-71ee-4da9-8d06-bdcb68513773\") " pod="openshift-monitoring/telemeter-client-56745b4487-g2wzx"
Jul 23 17:07:46.525899 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:07:46.525901    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d128b7fd-71ee-4da9-8d06-bdcb68513773-metrics-client-ca\") pod \"telemeter-client-56745b4487-g2wzx\" (UID: \"d128b7fd-71ee-4da9-8d06-bdcb68513773\") " pod="openshift-monitoring/telemeter-client-56745b4487-g2wzx"
Jul 23 17:07:46.526136 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:07:46.525930    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/d128b7fd-71ee-4da9-8d06-bdcb68513773-secret-telemeter-client\") pod \"telemeter-client-56745b4487-g2wzx\" (UID: \"d128b7fd-71ee-4da9-8d06-bdcb68513773\") " pod="openshift-monitoring/telemeter-client-56745b4487-g2wzx"
Jul 23 17:07:46.526136 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:07:46.525948    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/d128b7fd-71ee-4da9-8d06-bdcb68513773-telemeter-client-tls\") pod \"telemeter-client-56745b4487-g2wzx\" (UID: \"d128b7fd-71ee-4da9-8d06-bdcb68513773\") " pod="openshift-monitoring/telemeter-client-56745b4487-g2wzx"
Jul 23 17:07:46.526136 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:07:46.525977    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5x9sw\" (UniqueName: \"kubernetes.io/projected/d128b7fd-71ee-4da9-8d06-bdcb68513773-kube-api-access-5x9sw\") pod \"telemeter-client-56745b4487-g2wzx\" (UID: \"d128b7fd-71ee-4da9-8d06-bdcb68513773\") " pod="openshift-monitoring/telemeter-client-56745b4487-g2wzx"
Jul 23 17:07:46.526136 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:07:46.526016    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d128b7fd-71ee-4da9-8d06-bdcb68513773-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-56745b4487-g2wzx\" (UID: \"d128b7fd-71ee-4da9-8d06-bdcb68513773\") " pod="openshift-monitoring/telemeter-client-56745b4487-g2wzx"
Jul 23 17:07:46.526136 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:07:46.526052    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d128b7fd-71ee-4da9-8d06-bdcb68513773-telemeter-trusted-ca-bundle\") pod \"telemeter-client-56745b4487-g2wzx\" (UID: \"d128b7fd-71ee-4da9-8d06-bdcb68513773\") " pod="openshift-monitoring/telemeter-client-56745b4487-g2wzx"
Jul 23 17:07:46.526136 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:07:46.526094    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d128b7fd-71ee-4da9-8d06-bdcb68513773-serving-certs-ca-bundle\") pod \"telemeter-client-56745b4487-g2wzx\" (UID: \"d128b7fd-71ee-4da9-8d06-bdcb68513773\") " pod="openshift-monitoring/telemeter-client-56745b4487-g2wzx"
Jul 23 17:07:46.526756 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:07:46.526722    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d128b7fd-71ee-4da9-8d06-bdcb68513773-metrics-client-ca\") pod \"telemeter-client-56745b4487-g2wzx\" (UID: \"d128b7fd-71ee-4da9-8d06-bdcb68513773\") " pod="openshift-monitoring/telemeter-client-56745b4487-g2wzx"
Jul 23 17:07:46.526872 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:07:46.526756    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d128b7fd-71ee-4da9-8d06-bdcb68513773-serving-certs-ca-bundle\") pod \"telemeter-client-56745b4487-g2wzx\" (UID: \"d128b7fd-71ee-4da9-8d06-bdcb68513773\") " pod="openshift-monitoring/telemeter-client-56745b4487-g2wzx"
Jul 23 17:07:46.526985 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:07:46.526961    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d128b7fd-71ee-4da9-8d06-bdcb68513773-telemeter-trusted-ca-bundle\") pod \"telemeter-client-56745b4487-g2wzx\" (UID: \"d128b7fd-71ee-4da9-8d06-bdcb68513773\") " pod="openshift-monitoring/telemeter-client-56745b4487-g2wzx"
Jul 23 17:07:46.528632 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:07:46.528607    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d128b7fd-71ee-4da9-8d06-bdcb68513773-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-56745b4487-g2wzx\" (UID: \"d128b7fd-71ee-4da9-8d06-bdcb68513773\") " pod="openshift-monitoring/telemeter-client-56745b4487-g2wzx"
Jul 23 17:07:46.528719 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:07:46.528634    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/d128b7fd-71ee-4da9-8d06-bdcb68513773-telemeter-client-tls\") pod \"telemeter-client-56745b4487-g2wzx\" (UID: \"d128b7fd-71ee-4da9-8d06-bdcb68513773\") " pod="openshift-monitoring/telemeter-client-56745b4487-g2wzx"
Jul 23 17:07:46.528943 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:07:46.528921    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/d128b7fd-71ee-4da9-8d06-bdcb68513773-federate-client-tls\") pod \"telemeter-client-56745b4487-g2wzx\" (UID: \"d128b7fd-71ee-4da9-8d06-bdcb68513773\") " pod="openshift-monitoring/telemeter-client-56745b4487-g2wzx"
Jul 23 17:07:46.528982 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:07:46.528947    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/d128b7fd-71ee-4da9-8d06-bdcb68513773-secret-telemeter-client\") pod \"telemeter-client-56745b4487-g2wzx\" (UID: \"d128b7fd-71ee-4da9-8d06-bdcb68513773\") " pod="openshift-monitoring/telemeter-client-56745b4487-g2wzx"
Jul 23 17:07:46.534645 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:07:46.534624    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5x9sw\" (UniqueName: \"kubernetes.io/projected/d128b7fd-71ee-4da9-8d06-bdcb68513773-kube-api-access-5x9sw\") pod \"telemeter-client-56745b4487-g2wzx\" (UID: \"d128b7fd-71ee-4da9-8d06-bdcb68513773\") " pod="openshift-monitoring/telemeter-client-56745b4487-g2wzx"
Jul 23 17:07:46.628396 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:07:46.628340    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-56745b4487-g2wzx"
Jul 23 17:07:46.765603 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:07:46.765573    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-56745b4487-g2wzx"]
Jul 23 17:07:46.766160 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:07:46.766136    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd128b7fd_71ee_4da9_8d06_bdcb68513773.slice/crio-2a144a03a22ae1ba6076d054fab9203cfe837425f5c291dad02a39790cb8127d WatchSource:0}: Error finding container 2a144a03a22ae1ba6076d054fab9203cfe837425f5c291dad02a39790cb8127d: Status 404 returned error can't find the container with id 2a144a03a22ae1ba6076d054fab9203cfe837425f5c291dad02a39790cb8127d
Jul 23 17:07:47.447032 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:07:47.447000    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-56745b4487-g2wzx" event={"ID":"d128b7fd-71ee-4da9-8d06-bdcb68513773","Type":"ContainerStarted","Data":"2a144a03a22ae1ba6076d054fab9203cfe837425f5c291dad02a39790cb8127d"}
Jul 23 17:07:49.456305 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:07:49.456267    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-56745b4487-g2wzx" event={"ID":"d128b7fd-71ee-4da9-8d06-bdcb68513773","Type":"ContainerStarted","Data":"36df68be68ddd76d29b209fcba51a685fb1a7f888e9c2f00e2e1586a62102ce8"}
Jul 23 17:07:49.456696 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:07:49.456337    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-56745b4487-g2wzx" event={"ID":"d128b7fd-71ee-4da9-8d06-bdcb68513773","Type":"ContainerStarted","Data":"494c4b50a4b553467bf93d750827d8911c76d8ee9df64710134745971c124cdf"}
Jul 23 17:07:49.456696 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:07:49.456364    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-56745b4487-g2wzx" event={"ID":"d128b7fd-71ee-4da9-8d06-bdcb68513773","Type":"ContainerStarted","Data":"eb1a99602dafdc9cdb1fe98cfc7294ffb2865cd853bd2654ce2633cc56ef7df7"}
Jul 23 17:07:49.483653 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:07:49.483603    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-56745b4487-g2wzx" podStartSLOduration=1.264820094 podStartE2EDuration="3.483587347s" totalImagesPullingTime="2.218767253s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 17:07:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 17:07:49.481116427 +0000 UTC m=+165.251401946" watchObservedRunningTime="2026-07-23 17:07:49.483587347 +0000 UTC m=+165.253872866"
Jul 23 17:07:50.335157 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:07:50.335122    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:07:50.393276 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:07:50.393218    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:07:50.469389 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:07:50.469359    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:07:50.570150 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:07:50.570116    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:07:51.081239 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:07:51.081210    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:07:51.149210 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:07:51.149177    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:07:56.487235 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:07:56.487151    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:07:56.578778 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:07:56.578750    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:08:01.772320 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:08:01.772282    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:08:01.878240 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:08:01.878200    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:08:02.248730 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:08:02.248694    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:08:02.349124 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:08:02.349098    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:08:08.492577 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:08:08.492548    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:08:08.883417 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:08:08.883329    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:08:12.751837 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:08:12.751799    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:08:12.855220 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:08:12.855179    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:08:12.922136 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:08:12.922103    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:08:24.271229 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:08:24.271201    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:08:24.370760 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:08:24.370727    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:08:25.493030 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:08:25.492997    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:08:25.594277 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:08:25.594226    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:08:29.830510 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:08:29.830476    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:08:47.702847 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:08:47.702813    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:08:47.760520 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:08:47.760492    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:09:24.971784 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:09:24.971750    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:09:25.068780 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:09:25.068752    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:09:27.229093 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:09:27.229009    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404kxdx"]
Jul 23 17:09:27.231249 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:09:27.231233    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404kxdx"
Jul 23 17:09:27.233555 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:09:27.233531    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Jul 23 17:09:27.233854 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:09:27.233837    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Jul 23 17:09:27.234407 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:09:27.234391    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-4nc8x\""
Jul 23 17:09:27.241622 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:09:27.241600    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404kxdx"]
Jul 23 17:09:27.279481 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:09:27.279453    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nz6zm\" (UniqueName: \"kubernetes.io/projected/6901d914-673a-4afd-812f-749c1f367a89-kube-api-access-nz6zm\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404kxdx\" (UID: \"6901d914-673a-4afd-812f-749c1f367a89\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404kxdx"
Jul 23 17:09:27.279583 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:09:27.279501    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6901d914-673a-4afd-812f-749c1f367a89-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404kxdx\" (UID: \"6901d914-673a-4afd-812f-749c1f367a89\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404kxdx"
Jul 23 17:09:27.279583 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:09:27.279531    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6901d914-673a-4afd-812f-749c1f367a89-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404kxdx\" (UID: \"6901d914-673a-4afd-812f-749c1f367a89\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404kxdx"
Jul 23 17:09:27.379884 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:09:27.379857    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6901d914-673a-4afd-812f-749c1f367a89-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404kxdx\" (UID: \"6901d914-673a-4afd-812f-749c1f367a89\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404kxdx"
Jul 23 17:09:27.380026 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:09:27.379895    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nz6zm\" (UniqueName: \"kubernetes.io/projected/6901d914-673a-4afd-812f-749c1f367a89-kube-api-access-nz6zm\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404kxdx\" (UID: \"6901d914-673a-4afd-812f-749c1f367a89\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404kxdx"
Jul 23 17:09:27.380026 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:09:27.379959    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6901d914-673a-4afd-812f-749c1f367a89-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404kxdx\" (UID: \"6901d914-673a-4afd-812f-749c1f367a89\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404kxdx"
Jul 23 17:09:27.380333 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:09:27.380315    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6901d914-673a-4afd-812f-749c1f367a89-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404kxdx\" (UID: \"6901d914-673a-4afd-812f-749c1f367a89\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404kxdx"
Jul 23 17:09:27.380333 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:09:27.380324    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6901d914-673a-4afd-812f-749c1f367a89-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404kxdx\" (UID: \"6901d914-673a-4afd-812f-749c1f367a89\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404kxdx"
Jul 23 17:09:27.388690 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:09:27.388665    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nz6zm\" (UniqueName: \"kubernetes.io/projected/6901d914-673a-4afd-812f-749c1f367a89-kube-api-access-nz6zm\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404kxdx\" (UID: \"6901d914-673a-4afd-812f-749c1f367a89\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404kxdx"
Jul 23 17:09:27.539873 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:09:27.539789    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404kxdx"
Jul 23 17:09:27.668163 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:09:27.668135    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404kxdx"]
Jul 23 17:09:27.670596 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:09:27.670564    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6901d914_673a_4afd_812f_749c1f367a89.slice/crio-09b604a1131707d4b3a205c9c8e03010afdc47752c87193a479ef3394c2e2563 WatchSource:0}: Error finding container 09b604a1131707d4b3a205c9c8e03010afdc47752c87193a479ef3394c2e2563: Status 404 returned error can't find the container with id 09b604a1131707d4b3a205c9c8e03010afdc47752c87193a479ef3394c2e2563
Jul 23 17:09:27.811022 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:09:27.810943    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404kxdx" event={"ID":"6901d914-673a-4afd-812f-749c1f367a89","Type":"ContainerStarted","Data":"09b604a1131707d4b3a205c9c8e03010afdc47752c87193a479ef3394c2e2563"}
Jul 23 17:09:35.843402 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:09:35.843312    2577 generic.go:356] "Generic (PLEG): container finished" podID="6901d914-673a-4afd-812f-749c1f367a89" containerID="33efa666e6b1dffe4d0c017eddc6231c8ec0499780c4f4737da21b38cf4b2516" exitCode=0
Jul 23 17:09:35.843402 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:09:35.843354    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404kxdx" event={"ID":"6901d914-673a-4afd-812f-749c1f367a89","Type":"ContainerDied","Data":"33efa666e6b1dffe4d0c017eddc6231c8ec0499780c4f4737da21b38cf4b2516"}
Jul 23 17:09:38.861339 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:09:38.861304    2577 generic.go:356] "Generic (PLEG): container finished" podID="6901d914-673a-4afd-812f-749c1f367a89" containerID="c0d39355576d41844e69166c04f89b97f970289867dc6cc4113ebec7047ed25b" exitCode=0
Jul 23 17:09:38.861704 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:09:38.861367    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404kxdx" event={"ID":"6901d914-673a-4afd-812f-749c1f367a89","Type":"ContainerDied","Data":"c0d39355576d41844e69166c04f89b97f970289867dc6cc4113ebec7047ed25b"}
Jul 23 17:09:47.902401 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:09:47.902320    2577 generic.go:356] "Generic (PLEG): container finished" podID="6901d914-673a-4afd-812f-749c1f367a89" containerID="23d192b77b03b23fb12fe0e895ba11c95fa6feb14767c5d2d6f943c148049162" exitCode=0
Jul 23 17:09:47.902786 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:09:47.902405    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404kxdx" event={"ID":"6901d914-673a-4afd-812f-749c1f367a89","Type":"ContainerDied","Data":"23d192b77b03b23fb12fe0e895ba11c95fa6feb14767c5d2d6f943c148049162"}
Jul 23 17:09:49.054597 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:09:49.054569    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404kxdx"
Jul 23 17:09:49.159529 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:09:49.159463    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6901d914-673a-4afd-812f-749c1f367a89-bundle\") pod \"6901d914-673a-4afd-812f-749c1f367a89\" (UID: \"6901d914-673a-4afd-812f-749c1f367a89\") "
Jul 23 17:09:49.159663 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:09:49.159536    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nz6zm\" (UniqueName: \"kubernetes.io/projected/6901d914-673a-4afd-812f-749c1f367a89-kube-api-access-nz6zm\") pod \"6901d914-673a-4afd-812f-749c1f367a89\" (UID: \"6901d914-673a-4afd-812f-749c1f367a89\") "
Jul 23 17:09:49.159663 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:09:49.159561    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6901d914-673a-4afd-812f-749c1f367a89-util\") pod \"6901d914-673a-4afd-812f-749c1f367a89\" (UID: \"6901d914-673a-4afd-812f-749c1f367a89\") "
Jul 23 17:09:49.160155 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:09:49.160125    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6901d914-673a-4afd-812f-749c1f367a89-bundle" (OuterVolumeSpecName: "bundle") pod "6901d914-673a-4afd-812f-749c1f367a89" (UID: "6901d914-673a-4afd-812f-749c1f367a89"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 17:09:49.161590 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:09:49.161571    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6901d914-673a-4afd-812f-749c1f367a89-kube-api-access-nz6zm" (OuterVolumeSpecName: "kube-api-access-nz6zm") pod "6901d914-673a-4afd-812f-749c1f367a89" (UID: "6901d914-673a-4afd-812f-749c1f367a89"). InnerVolumeSpecName "kube-api-access-nz6zm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 17:09:49.163977 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:09:49.163955    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6901d914-673a-4afd-812f-749c1f367a89-util" (OuterVolumeSpecName: "util") pod "6901d914-673a-4afd-812f-749c1f367a89" (UID: "6901d914-673a-4afd-812f-749c1f367a89"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 17:09:49.261044 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:09:49.261015    2577 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6901d914-673a-4afd-812f-749c1f367a89-bundle\") on node \"ip-10-0-129-216.ec2.internal\" DevicePath \"\""
Jul 23 17:09:49.261044 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:09:49.261042    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nz6zm\" (UniqueName: \"kubernetes.io/projected/6901d914-673a-4afd-812f-749c1f367a89-kube-api-access-nz6zm\") on node \"ip-10-0-129-216.ec2.internal\" DevicePath \"\""
Jul 23 17:09:49.261044 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:09:49.261052    2577 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6901d914-673a-4afd-812f-749c1f367a89-util\") on node \"ip-10-0-129-216.ec2.internal\" DevicePath \"\""
Jul 23 17:09:49.911876 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:09:49.911849    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404kxdx"
Jul 23 17:09:49.912054 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:09:49.911846    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675404kxdx" event={"ID":"6901d914-673a-4afd-812f-749c1f367a89","Type":"ContainerDied","Data":"09b604a1131707d4b3a205c9c8e03010afdc47752c87193a479ef3394c2e2563"}
Jul 23 17:09:49.912054 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:09:49.911976    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="09b604a1131707d4b3a205c9c8e03010afdc47752c87193a479ef3394c2e2563"
Jul 23 17:09:53.301717 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:09:53.301648    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:09:55.200168 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:09:55.200137    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:09:55.377490 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:09:55.377455    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-ts88g"]
Jul 23 17:09:55.377749 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:09:55.377738    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6901d914-673a-4afd-812f-749c1f367a89" containerName="util"
Jul 23 17:09:55.377791 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:09:55.377751    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="6901d914-673a-4afd-812f-749c1f367a89" containerName="util"
Jul 23 17:09:55.377791 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:09:55.377757    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6901d914-673a-4afd-812f-749c1f367a89" containerName="extract"
Jul 23 17:09:55.377791 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:09:55.377762    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="6901d914-673a-4afd-812f-749c1f367a89" containerName="extract"
Jul 23 17:09:55.377791 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:09:55.377772    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6901d914-673a-4afd-812f-749c1f367a89" containerName="pull"
Jul 23 17:09:55.377791 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:09:55.377778    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="6901d914-673a-4afd-812f-749c1f367a89" containerName="pull"
Jul 23 17:09:55.377939 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:09:55.377829    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="6901d914-673a-4afd-812f-749c1f367a89" containerName="extract"
Jul 23 17:09:55.382512 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:09:55.382497    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-ts88g"
Jul 23 17:09:55.385598 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:09:55.385577    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\""
Jul 23 17:09:55.385911 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:09:55.385891    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-fsgvv\""
Jul 23 17:09:55.385992 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:09:55.385939    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\""
Jul 23 17:09:55.397845 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:09:55.397820    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-ts88g"]
Jul 23 17:09:55.508217 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:09:55.508137    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1f9067a4-4309-4788-b365-1c3d09c00074-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-ts88g\" (UID: \"1f9067a4-4309-4788-b365-1c3d09c00074\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-ts88g"
Jul 23 17:09:55.508217 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:09:55.508197    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84cnq\" (UniqueName: \"kubernetes.io/projected/1f9067a4-4309-4788-b365-1c3d09c00074-kube-api-access-84cnq\") pod \"cert-manager-operator-controller-manager-55c444bf86-ts88g\" (UID: \"1f9067a4-4309-4788-b365-1c3d09c00074\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-ts88g"
Jul 23 17:09:55.609136 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:09:55.609100    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1f9067a4-4309-4788-b365-1c3d09c00074-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-ts88g\" (UID: \"1f9067a4-4309-4788-b365-1c3d09c00074\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-ts88g"
Jul 23 17:09:55.609305 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:09:55.609166    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84cnq\" (UniqueName: \"kubernetes.io/projected/1f9067a4-4309-4788-b365-1c3d09c00074-kube-api-access-84cnq\") pod \"cert-manager-operator-controller-manager-55c444bf86-ts88g\" (UID: \"1f9067a4-4309-4788-b365-1c3d09c00074\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-ts88g"
Jul 23 17:09:55.609518 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:09:55.609495    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1f9067a4-4309-4788-b365-1c3d09c00074-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-ts88g\" (UID: \"1f9067a4-4309-4788-b365-1c3d09c00074\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-ts88g"
Jul 23 17:09:55.619179 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:09:55.619156    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-84cnq\" (UniqueName: \"kubernetes.io/projected/1f9067a4-4309-4788-b365-1c3d09c00074-kube-api-access-84cnq\") pod \"cert-manager-operator-controller-manager-55c444bf86-ts88g\" (UID: \"1f9067a4-4309-4788-b365-1c3d09c00074\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-ts88g"
Jul 23 17:09:55.690880 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:09:55.690844    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-ts88g"
Jul 23 17:09:55.823156 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:09:55.823084    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-ts88g"]
Jul 23 17:09:55.828723 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:09:55.828694    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f9067a4_4309_4788_b365_1c3d09c00074.slice/crio-b1260e0ff3e199159ee8b60d5d911f8314b5e1af373e41d1876a9c6539f9b99c WatchSource:0}: Error finding container b1260e0ff3e199159ee8b60d5d911f8314b5e1af373e41d1876a9c6539f9b99c: Status 404 returned error can't find the container with id b1260e0ff3e199159ee8b60d5d911f8314b5e1af373e41d1876a9c6539f9b99c
Jul 23 17:09:55.935040 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:09:55.935001    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-ts88g" event={"ID":"1f9067a4-4309-4788-b365-1c3d09c00074","Type":"ContainerStarted","Data":"b1260e0ff3e199159ee8b60d5d911f8314b5e1af373e41d1876a9c6539f9b99c"}
Jul 23 17:09:56.946560 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:09:56.946521    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:09:57.054137 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:09:57.054110    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:09:57.949891 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:09:57.949855    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-ts88g" event={"ID":"1f9067a4-4309-4788-b365-1c3d09c00074","Type":"ContainerStarted","Data":"2274e4a70443cfe469251b1b4c15fe83369b414d25b9fb5e3ac222c752af2338"}
Jul 23 17:09:57.977995 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:09:57.977946    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-ts88g" podStartSLOduration=1.337348727 podStartE2EDuration="2.977930164s" totalImagesPullingTime="1.640581437s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 17:09:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 17:09:57.976108411 +0000 UTC m=+293.746393930" watchObservedRunningTime="2026-07-23 17:09:57.977930164 +0000 UTC m=+293.748215679"
Jul 23 17:09:58.395541 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:09:58.395464    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:09:58.537355 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:09:58.537318    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:09:59.112581 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:09:59.112542    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhjj7v"]
Jul 23 17:09:59.115095 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:09:59.115079    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhjj7v"
Jul 23 17:09:59.118536 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:09:59.118515    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-4nc8x\""
Jul 23 17:09:59.118536 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:09:59.118527    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Jul 23 17:09:59.119414 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:09:59.119395    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Jul 23 17:09:59.130282 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:09:59.130244    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhjj7v"]
Jul 23 17:09:59.235514 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:09:59.235481    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2001aa1e-f3b0-4929-853a-4ccd0ed54da5-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhjj7v\" (UID: \"2001aa1e-f3b0-4929-853a-4ccd0ed54da5\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhjj7v"
Jul 23 17:09:59.235678 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:09:59.235540    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2001aa1e-f3b0-4929-853a-4ccd0ed54da5-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhjj7v\" (UID: \"2001aa1e-f3b0-4929-853a-4ccd0ed54da5\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhjj7v"
Jul 23 17:09:59.235678 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:09:59.235595    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fprgt\" (UniqueName: \"kubernetes.io/projected/2001aa1e-f3b0-4929-853a-4ccd0ed54da5-kube-api-access-fprgt\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhjj7v\" (UID: \"2001aa1e-f3b0-4929-853a-4ccd0ed54da5\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhjj7v"
Jul 23 17:09:59.304670 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:09:59.304638    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:09:59.336180 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:09:59.336153    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2001aa1e-f3b0-4929-853a-4ccd0ed54da5-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhjj7v\" (UID: \"2001aa1e-f3b0-4929-853a-4ccd0ed54da5\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhjj7v"
Jul 23 17:09:59.336331 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:09:59.336195    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fprgt\" (UniqueName: \"kubernetes.io/projected/2001aa1e-f3b0-4929-853a-4ccd0ed54da5-kube-api-access-fprgt\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhjj7v\" (UID: \"2001aa1e-f3b0-4929-853a-4ccd0ed54da5\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhjj7v"
Jul 23 17:09:59.336331 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:09:59.336232    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2001aa1e-f3b0-4929-853a-4ccd0ed54da5-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhjj7v\" (UID: \"2001aa1e-f3b0-4929-853a-4ccd0ed54da5\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhjj7v"
Jul 23 17:09:59.336544 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:09:59.336524    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2001aa1e-f3b0-4929-853a-4ccd0ed54da5-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhjj7v\" (UID: \"2001aa1e-f3b0-4929-853a-4ccd0ed54da5\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhjj7v"
Jul 23 17:09:59.336600 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:09:59.336559    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2001aa1e-f3b0-4929-853a-4ccd0ed54da5-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhjj7v\" (UID: \"2001aa1e-f3b0-4929-853a-4ccd0ed54da5\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhjj7v"
Jul 23 17:09:59.344765 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:09:59.344736    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fprgt\" (UniqueName: \"kubernetes.io/projected/2001aa1e-f3b0-4929-853a-4ccd0ed54da5-kube-api-access-fprgt\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhjj7v\" (UID: \"2001aa1e-f3b0-4929-853a-4ccd0ed54da5\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhjj7v"
Jul 23 17:09:59.421544 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:09:59.421449    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:09:59.423388 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:09:59.423370    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhjj7v"
Jul 23 17:09:59.568617 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:09:59.568589    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhjj7v"]
Jul 23 17:09:59.570871 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:09:59.570844    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2001aa1e_f3b0_4929_853a_4ccd0ed54da5.slice/crio-5b68becbf57d6bb207031e337e4a2949a91dbeb1b955ebe82353f14a20a85a08 WatchSource:0}: Error finding container 5b68becbf57d6bb207031e337e4a2949a91dbeb1b955ebe82353f14a20a85a08: Status 404 returned error can't find the container with id 5b68becbf57d6bb207031e337e4a2949a91dbeb1b955ebe82353f14a20a85a08
Jul 23 17:09:59.958856 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:09:59.958823    2577 generic.go:356] "Generic (PLEG): container finished" podID="2001aa1e-f3b0-4929-853a-4ccd0ed54da5" containerID="da98703e3301c6305e2baaee6983a69cabf5f7a2e80870f8d4d79214cda1d9d2" exitCode=0
Jul 23 17:09:59.959036 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:09:59.958916    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhjj7v" event={"ID":"2001aa1e-f3b0-4929-853a-4ccd0ed54da5","Type":"ContainerDied","Data":"da98703e3301c6305e2baaee6983a69cabf5f7a2e80870f8d4d79214cda1d9d2"}
Jul 23 17:09:59.959036 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:09:59.958976    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhjj7v" event={"ID":"2001aa1e-f3b0-4929-853a-4ccd0ed54da5","Type":"ContainerStarted","Data":"5b68becbf57d6bb207031e337e4a2949a91dbeb1b955ebe82353f14a20a85a08"}
Jul 23 17:10:00.295098 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:00.295017    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-wnxh4"]
Jul 23 17:10:00.297277 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:00.297243    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-wnxh4"
Jul 23 17:10:00.299726 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:00.299694    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Jul 23 17:10:00.299857 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:00.299814    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-2qpht\""
Jul 23 17:10:00.299926 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:00.299895    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Jul 23 17:10:00.306965 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:00.306946    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-wnxh4"]
Jul 23 17:10:00.444707 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:00.444678    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4189e772-b973-4981-bd63-c6379f74b1c1-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-wnxh4\" (UID: \"4189e772-b973-4981-bd63-c6379f74b1c1\") " pod="cert-manager/cert-manager-webhook-6486ffd997-wnxh4"
Jul 23 17:10:00.444865 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:00.444726    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hfz4q\" (UniqueName: \"kubernetes.io/projected/4189e772-b973-4981-bd63-c6379f74b1c1-kube-api-access-hfz4q\") pod \"cert-manager-webhook-6486ffd997-wnxh4\" (UID: \"4189e772-b973-4981-bd63-c6379f74b1c1\") " pod="cert-manager/cert-manager-webhook-6486ffd997-wnxh4"
Jul 23 17:10:00.545810 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:00.545733    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4189e772-b973-4981-bd63-c6379f74b1c1-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-wnxh4\" (UID: \"4189e772-b973-4981-bd63-c6379f74b1c1\") " pod="cert-manager/cert-manager-webhook-6486ffd997-wnxh4"
Jul 23 17:10:00.545810 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:00.545785    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hfz4q\" (UniqueName: \"kubernetes.io/projected/4189e772-b973-4981-bd63-c6379f74b1c1-kube-api-access-hfz4q\") pod \"cert-manager-webhook-6486ffd997-wnxh4\" (UID: \"4189e772-b973-4981-bd63-c6379f74b1c1\") " pod="cert-manager/cert-manager-webhook-6486ffd997-wnxh4"
Jul 23 17:10:00.555123 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:00.555098    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4189e772-b973-4981-bd63-c6379f74b1c1-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-wnxh4\" (UID: \"4189e772-b973-4981-bd63-c6379f74b1c1\") " pod="cert-manager/cert-manager-webhook-6486ffd997-wnxh4"
Jul 23 17:10:00.555382 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:00.555357    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hfz4q\" (UniqueName: \"kubernetes.io/projected/4189e772-b973-4981-bd63-c6379f74b1c1-kube-api-access-hfz4q\") pod \"cert-manager-webhook-6486ffd997-wnxh4\" (UID: \"4189e772-b973-4981-bd63-c6379f74b1c1\") " pod="cert-manager/cert-manager-webhook-6486ffd997-wnxh4"
Jul 23 17:10:00.615797 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:00.615757    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-wnxh4"
Jul 23 17:10:00.769455 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:00.769432    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-wnxh4"]
Jul 23 17:10:00.772191 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:10:00.772160    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4189e772_b973_4981_bd63_c6379f74b1c1.slice/crio-f74448aaa3911146f198c5e70c784854692dabc9b2d41e35dcf44e5e33e4936b WatchSource:0}: Error finding container f74448aaa3911146f198c5e70c784854692dabc9b2d41e35dcf44e5e33e4936b: Status 404 returned error can't find the container with id f74448aaa3911146f198c5e70c784854692dabc9b2d41e35dcf44e5e33e4936b
Jul 23 17:10:00.965317 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:00.965278    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-wnxh4" event={"ID":"4189e772-b973-4981-bd63-c6379f74b1c1","Type":"ContainerStarted","Data":"f74448aaa3911146f198c5e70c784854692dabc9b2d41e35dcf44e5e33e4936b"}
Jul 23 17:10:01.299271 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:01.299177    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:10:01.707332 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:01.707285    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:10:01.976014 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:01.975982    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhjj7v" event={"ID":"2001aa1e-f3b0-4929-853a-4ccd0ed54da5","Type":"ContainerStarted","Data":"c46daee434c4b0c9fadcf5ededa3775d93b81c07e6569e66d4c2af0f097db7ca"}
Jul 23 17:10:02.256382 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:02.256296    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-kdz7k"]
Jul 23 17:10:02.258923 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:02.258902    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-kdz7k"
Jul 23 17:10:02.261229 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:02.261209    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-nw5sf\""
Jul 23 17:10:02.266743 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:02.266720    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-kdz7k"]
Jul 23 17:10:02.363961 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:02.363928    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5baf5fff-9f4f-4411-9c90-8482a647b6a4-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-kdz7k\" (UID: \"5baf5fff-9f4f-4411-9c90-8482a647b6a4\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-kdz7k"
Jul 23 17:10:02.363961 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:02.363964    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g585s\" (UniqueName: \"kubernetes.io/projected/5baf5fff-9f4f-4411-9c90-8482a647b6a4-kube-api-access-g585s\") pod \"cert-manager-cainjector-559cdd8c87-kdz7k\" (UID: \"5baf5fff-9f4f-4411-9c90-8482a647b6a4\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-kdz7k"
Jul 23 17:10:02.465212 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:02.465175    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5baf5fff-9f4f-4411-9c90-8482a647b6a4-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-kdz7k\" (UID: \"5baf5fff-9f4f-4411-9c90-8482a647b6a4\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-kdz7k"
Jul 23 17:10:02.465395 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:02.465229    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g585s\" (UniqueName: \"kubernetes.io/projected/5baf5fff-9f4f-4411-9c90-8482a647b6a4-kube-api-access-g585s\") pod \"cert-manager-cainjector-559cdd8c87-kdz7k\" (UID: \"5baf5fff-9f4f-4411-9c90-8482a647b6a4\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-kdz7k"
Jul 23 17:10:02.476427 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:02.476361    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5baf5fff-9f4f-4411-9c90-8482a647b6a4-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-kdz7k\" (UID: \"5baf5fff-9f4f-4411-9c90-8482a647b6a4\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-kdz7k"
Jul 23 17:10:02.476691 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:02.476672    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g585s\" (UniqueName: \"kubernetes.io/projected/5baf5fff-9f4f-4411-9c90-8482a647b6a4-kube-api-access-g585s\") pod \"cert-manager-cainjector-559cdd8c87-kdz7k\" (UID: \"5baf5fff-9f4f-4411-9c90-8482a647b6a4\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-kdz7k"
Jul 23 17:10:02.567963 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:02.567880    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-kdz7k"
Jul 23 17:10:02.719730 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:02.719701    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-kdz7k"]
Jul 23 17:10:02.722938 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:10:02.722910    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5baf5fff_9f4f_4411_9c90_8482a647b6a4.slice/crio-ac6a5019fe3ce85b8f7319c8fe286892c204abd068772c85aeddf466eb9f1440 WatchSource:0}: Error finding container ac6a5019fe3ce85b8f7319c8fe286892c204abd068772c85aeddf466eb9f1440: Status 404 returned error can't find the container with id ac6a5019fe3ce85b8f7319c8fe286892c204abd068772c85aeddf466eb9f1440
Jul 23 17:10:02.982950 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:02.982911    2577 generic.go:356] "Generic (PLEG): container finished" podID="2001aa1e-f3b0-4929-853a-4ccd0ed54da5" containerID="c46daee434c4b0c9fadcf5ededa3775d93b81c07e6569e66d4c2af0f097db7ca" exitCode=0
Jul 23 17:10:02.982950 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:02.982948    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhjj7v" event={"ID":"2001aa1e-f3b0-4929-853a-4ccd0ed54da5","Type":"ContainerDied","Data":"c46daee434c4b0c9fadcf5ededa3775d93b81c07e6569e66d4c2af0f097db7ca"}
Jul 23 17:10:02.984415 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:02.984387    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-kdz7k" event={"ID":"5baf5fff-9f4f-4411-9c90-8482a647b6a4","Type":"ContainerStarted","Data":"ac6a5019fe3ce85b8f7319c8fe286892c204abd068772c85aeddf466eb9f1440"}
Jul 23 17:10:03.503747 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:03.503707    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:10:03.573651 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:03.573621    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:10:03.991409 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:03.991372    2577 generic.go:356] "Generic (PLEG): container finished" podID="2001aa1e-f3b0-4929-853a-4ccd0ed54da5" containerID="1893b5a09da37f33787b734b7196377211eccd71d90f1f043b4bfe755ed7a060" exitCode=0
Jul 23 17:10:03.991692 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:03.991460    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhjj7v" event={"ID":"2001aa1e-f3b0-4929-853a-4ccd0ed54da5","Type":"ContainerDied","Data":"1893b5a09da37f33787b734b7196377211eccd71d90f1f043b4bfe755ed7a060"}
Jul 23 17:10:04.605647 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:04.605582    2577 kubelet.go:1648] "Image garbage collection succeeded"
Jul 23 17:10:04.996754 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:04.996724    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-wnxh4" event={"ID":"4189e772-b973-4981-bd63-c6379f74b1c1","Type":"ContainerStarted","Data":"1491c3ba8202e961430c0fff24be506816ebb5d6570c22b3c73dd05cd089192b"}
Jul 23 17:10:04.997035 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:04.997014    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-wnxh4"
Jul 23 17:10:05.013567 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:05.013510    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-wnxh4" podStartSLOduration=1.461990292 podStartE2EDuration="5.013491555s" totalImagesPullingTime="3.551501263s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 17:10:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 17:10:05.012780159 +0000 UTC m=+300.783065679" watchObservedRunningTime="2026-07-23 17:10:05.013491555 +0000 UTC m=+300.783777075"
Jul 23 17:10:05.127139 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:05.127112    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhjj7v"
Jul 23 17:10:05.292809 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:05.292731    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2001aa1e-f3b0-4929-853a-4ccd0ed54da5-util\") pod \"2001aa1e-f3b0-4929-853a-4ccd0ed54da5\" (UID: \"2001aa1e-f3b0-4929-853a-4ccd0ed54da5\") "
Jul 23 17:10:05.292809 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:05.292767    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2001aa1e-f3b0-4929-853a-4ccd0ed54da5-bundle\") pod \"2001aa1e-f3b0-4929-853a-4ccd0ed54da5\" (UID: \"2001aa1e-f3b0-4929-853a-4ccd0ed54da5\") "
Jul 23 17:10:05.292809 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:05.292805    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fprgt\" (UniqueName: \"kubernetes.io/projected/2001aa1e-f3b0-4929-853a-4ccd0ed54da5-kube-api-access-fprgt\") pod \"2001aa1e-f3b0-4929-853a-4ccd0ed54da5\" (UID: \"2001aa1e-f3b0-4929-853a-4ccd0ed54da5\") "
Jul 23 17:10:05.293170 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:05.293146    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2001aa1e-f3b0-4929-853a-4ccd0ed54da5-bundle" (OuterVolumeSpecName: "bundle") pod "2001aa1e-f3b0-4929-853a-4ccd0ed54da5" (UID: "2001aa1e-f3b0-4929-853a-4ccd0ed54da5"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 17:10:05.295196 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:05.295167    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2001aa1e-f3b0-4929-853a-4ccd0ed54da5-kube-api-access-fprgt" (OuterVolumeSpecName: "kube-api-access-fprgt") pod "2001aa1e-f3b0-4929-853a-4ccd0ed54da5" (UID: "2001aa1e-f3b0-4929-853a-4ccd0ed54da5"). InnerVolumeSpecName "kube-api-access-fprgt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 17:10:05.298629 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:05.298588    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2001aa1e-f3b0-4929-853a-4ccd0ed54da5-util" (OuterVolumeSpecName: "util") pod "2001aa1e-f3b0-4929-853a-4ccd0ed54da5" (UID: "2001aa1e-f3b0-4929-853a-4ccd0ed54da5"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 17:10:05.393797 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:05.393737    2577 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2001aa1e-f3b0-4929-853a-4ccd0ed54da5-util\") on node \"ip-10-0-129-216.ec2.internal\" DevicePath \"\""
Jul 23 17:10:05.393797 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:05.393762    2577 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2001aa1e-f3b0-4929-853a-4ccd0ed54da5-bundle\") on node \"ip-10-0-129-216.ec2.internal\" DevicePath \"\""
Jul 23 17:10:05.393797 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:05.393771    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fprgt\" (UniqueName: \"kubernetes.io/projected/2001aa1e-f3b0-4929-853a-4ccd0ed54da5-kube-api-access-fprgt\") on node \"ip-10-0-129-216.ec2.internal\" DevicePath \"\""
Jul 23 17:10:05.729778 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:05.729743    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:10:05.829963 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:05.829928    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:10:06.002169 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:06.002078    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-kdz7k" event={"ID":"5baf5fff-9f4f-4411-9c90-8482a647b6a4","Type":"ContainerStarted","Data":"a7b081a4292f0aee84bb256965d2475732bd231f9ecbc4f11682c7d47586f1a2"}
Jul 23 17:10:06.004010 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:06.003975    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhjj7v" event={"ID":"2001aa1e-f3b0-4929-853a-4ccd0ed54da5","Type":"ContainerDied","Data":"5b68becbf57d6bb207031e337e4a2949a91dbeb1b955ebe82353f14a20a85a08"}
Jul 23 17:10:06.004139 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:06.004025    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5b68becbf57d6bb207031e337e4a2949a91dbeb1b955ebe82353f14a20a85a08"
Jul 23 17:10:06.004139 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:06.004082    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhjj7v"
Jul 23 17:10:06.017815 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:06.017773    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-kdz7k" podStartSLOduration=1.792506457 podStartE2EDuration="4.017758949s" totalImagesPullingTime="2.225252492s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 17:10:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 17:10:06.017151214 +0000 UTC m=+301.787436744" watchObservedRunningTime="2026-07-23 17:10:06.017758949 +0000 UTC m=+301.788044469"
Jul 23 17:10:09.908823 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:09.908786    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:10:10.765533 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:10.765506    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-pzjzf"]
Jul 23 17:10:10.765784 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:10.765773    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2001aa1e-f3b0-4929-853a-4ccd0ed54da5" containerName="extract"
Jul 23 17:10:10.765828 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:10.765786    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="2001aa1e-f3b0-4929-853a-4ccd0ed54da5" containerName="extract"
Jul 23 17:10:10.765828 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:10.765803    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2001aa1e-f3b0-4929-853a-4ccd0ed54da5" containerName="pull"
Jul 23 17:10:10.765828 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:10.765809    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="2001aa1e-f3b0-4929-853a-4ccd0ed54da5" containerName="pull"
Jul 23 17:10:10.765828 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:10.765816    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2001aa1e-f3b0-4929-853a-4ccd0ed54da5" containerName="util"
Jul 23 17:10:10.765828 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:10.765821    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="2001aa1e-f3b0-4929-853a-4ccd0ed54da5" containerName="util"
Jul 23 17:10:10.765975 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:10.765870    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="2001aa1e-f3b0-4929-853a-4ccd0ed54da5" containerName="extract"
Jul 23 17:10:10.769000 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:10.768985    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-pzjzf"
Jul 23 17:10:10.771425 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:10.771406    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-s7ts2\""
Jul 23 17:10:10.773702 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:10.773681    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-pzjzf"]
Jul 23 17:10:10.939310 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:10.939210    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9kjpr\" (UniqueName: \"kubernetes.io/projected/9ef2eeb3-2d65-455f-81f7-a032c22b6833-kube-api-access-9kjpr\") pod \"cert-manager-55b4d954d5-pzjzf\" (UID: \"9ef2eeb3-2d65-455f-81f7-a032c22b6833\") " pod="cert-manager/cert-manager-55b4d954d5-pzjzf"
Jul 23 17:10:10.939310 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:10.939250    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9ef2eeb3-2d65-455f-81f7-a032c22b6833-bound-sa-token\") pod \"cert-manager-55b4d954d5-pzjzf\" (UID: \"9ef2eeb3-2d65-455f-81f7-a032c22b6833\") " pod="cert-manager/cert-manager-55b4d954d5-pzjzf"
Jul 23 17:10:11.005948 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:11.005914    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-wnxh4"
Jul 23 17:10:11.040265 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:11.040211    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9kjpr\" (UniqueName: \"kubernetes.io/projected/9ef2eeb3-2d65-455f-81f7-a032c22b6833-kube-api-access-9kjpr\") pod \"cert-manager-55b4d954d5-pzjzf\" (UID: \"9ef2eeb3-2d65-455f-81f7-a032c22b6833\") " pod="cert-manager/cert-manager-55b4d954d5-pzjzf"
Jul 23 17:10:11.040406 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:11.040298    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9ef2eeb3-2d65-455f-81f7-a032c22b6833-bound-sa-token\") pod \"cert-manager-55b4d954d5-pzjzf\" (UID: \"9ef2eeb3-2d65-455f-81f7-a032c22b6833\") " pod="cert-manager/cert-manager-55b4d954d5-pzjzf"
Jul 23 17:10:11.051056 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:11.051025    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9ef2eeb3-2d65-455f-81f7-a032c22b6833-bound-sa-token\") pod \"cert-manager-55b4d954d5-pzjzf\" (UID: \"9ef2eeb3-2d65-455f-81f7-a032c22b6833\") " pod="cert-manager/cert-manager-55b4d954d5-pzjzf"
Jul 23 17:10:11.051404 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:11.051383    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9kjpr\" (UniqueName: \"kubernetes.io/projected/9ef2eeb3-2d65-455f-81f7-a032c22b6833-kube-api-access-9kjpr\") pod \"cert-manager-55b4d954d5-pzjzf\" (UID: \"9ef2eeb3-2d65-455f-81f7-a032c22b6833\") " pod="cert-manager/cert-manager-55b4d954d5-pzjzf"
Jul 23 17:10:11.078471 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:11.078446    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-pzjzf"
Jul 23 17:10:11.223729 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:11.223706    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-pzjzf"]
Jul 23 17:10:11.225999 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:10:11.225973    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9ef2eeb3_2d65_455f_81f7_a032c22b6833.slice/crio-7fc198d6a74722b3087bf8a29d8c68884c8786232bea8e197bff9d986df69a4b WatchSource:0}: Error finding container 7fc198d6a74722b3087bf8a29d8c68884c8786232bea8e197bff9d986df69a4b: Status 404 returned error can't find the container with id 7fc198d6a74722b3087bf8a29d8c68884c8786232bea8e197bff9d986df69a4b
Jul 23 17:10:12.031520 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:12.031482    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-pzjzf" event={"ID":"9ef2eeb3-2d65-455f-81f7-a032c22b6833","Type":"ContainerStarted","Data":"b33e7b2485cbb3180a514ed16c599976a026659c198f504deb6e0b858b254a95"}
Jul 23 17:10:12.031971 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:12.031529    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-pzjzf" event={"ID":"9ef2eeb3-2d65-455f-81f7-a032c22b6833","Type":"ContainerStarted","Data":"7fc198d6a74722b3087bf8a29d8c68884c8786232bea8e197bff9d986df69a4b"}
Jul 23 17:10:12.053578 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:12.053529    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-pzjzf" podStartSLOduration=2.053514954 podStartE2EDuration="2.053514954s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 17:10:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 17:10:12.051158614 +0000 UTC m=+307.821444171" watchObservedRunningTime="2026-07-23 17:10:12.053514954 +0000 UTC m=+307.823800473"
Jul 23 17:10:14.808861 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:14.808826    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:10:14.909045 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:14.909013    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:10:21.080073 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:21.080031    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:10:21.962464 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:21.962427    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:10:22.773086 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:22.773060    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:10:23.416714 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:23.416678    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af7qqbp"]
Jul 23 17:10:23.421537 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:23.421522    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af7qqbp"
Jul 23 17:10:23.424372 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:23.424354    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Jul 23 17:10:23.424467 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:23.424425    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-4nc8x\""
Jul 23 17:10:23.424467 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:23.424426    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Jul 23 17:10:23.429235 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:23.429215    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af7qqbp"]
Jul 23 17:10:23.439122 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:23.439095    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwfhf\" (UniqueName: \"kubernetes.io/projected/b272f0fc-34de-4752-a5b2-a1591ca98bf0-kube-api-access-wwfhf\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af7qqbp\" (UID: \"b272f0fc-34de-4752-a5b2-a1591ca98bf0\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af7qqbp"
Jul 23 17:10:23.439238 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:23.439150    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b272f0fc-34de-4752-a5b2-a1591ca98bf0-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af7qqbp\" (UID: \"b272f0fc-34de-4752-a5b2-a1591ca98bf0\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af7qqbp"
Jul 23 17:10:23.439238 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:23.439216    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b272f0fc-34de-4752-a5b2-a1591ca98bf0-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af7qqbp\" (UID: \"b272f0fc-34de-4752-a5b2-a1591ca98bf0\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af7qqbp"
Jul 23 17:10:23.540629 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:23.540592    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwfhf\" (UniqueName: \"kubernetes.io/projected/b272f0fc-34de-4752-a5b2-a1591ca98bf0-kube-api-access-wwfhf\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af7qqbp\" (UID: \"b272f0fc-34de-4752-a5b2-a1591ca98bf0\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af7qqbp"
Jul 23 17:10:23.540794 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:23.540649    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b272f0fc-34de-4752-a5b2-a1591ca98bf0-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af7qqbp\" (UID: \"b272f0fc-34de-4752-a5b2-a1591ca98bf0\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af7qqbp"
Jul 23 17:10:23.540794 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:23.540692    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b272f0fc-34de-4752-a5b2-a1591ca98bf0-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af7qqbp\" (UID: \"b272f0fc-34de-4752-a5b2-a1591ca98bf0\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af7qqbp"
Jul 23 17:10:23.541049 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:23.541032    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b272f0fc-34de-4752-a5b2-a1591ca98bf0-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af7qqbp\" (UID: \"b272f0fc-34de-4752-a5b2-a1591ca98bf0\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af7qqbp"
Jul 23 17:10:23.541085 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:23.541062    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b272f0fc-34de-4752-a5b2-a1591ca98bf0-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af7qqbp\" (UID: \"b272f0fc-34de-4752-a5b2-a1591ca98bf0\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af7qqbp"
Jul 23 17:10:23.548962 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:23.548931    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwfhf\" (UniqueName: \"kubernetes.io/projected/b272f0fc-34de-4752-a5b2-a1591ca98bf0-kube-api-access-wwfhf\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af7qqbp\" (UID: \"b272f0fc-34de-4752-a5b2-a1591ca98bf0\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af7qqbp"
Jul 23 17:10:23.731579 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:23.731553    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af7qqbp"
Jul 23 17:10:23.862234 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:23.862203    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af7qqbp"]
Jul 23 17:10:23.863206 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:10:23.863182    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb272f0fc_34de_4752_a5b2_a1591ca98bf0.slice/crio-71ec3b95183bc3cbdf9037d49de060976f49cb553d397c9eb5fa50743026b622 WatchSource:0}: Error finding container 71ec3b95183bc3cbdf9037d49de060976f49cb553d397c9eb5fa50743026b622: Status 404 returned error can't find the container with id 71ec3b95183bc3cbdf9037d49de060976f49cb553d397c9eb5fa50743026b622
Jul 23 17:10:24.085216 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:24.085123    2577 generic.go:356] "Generic (PLEG): container finished" podID="b272f0fc-34de-4752-a5b2-a1591ca98bf0" containerID="a6a9b4da6d35514f26432985948a8e10fe966f7d69c0a338d6f1873c6a8035aa" exitCode=0
Jul 23 17:10:24.085216 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:24.085196    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af7qqbp" event={"ID":"b272f0fc-34de-4752-a5b2-a1591ca98bf0","Type":"ContainerDied","Data":"a6a9b4da6d35514f26432985948a8e10fe966f7d69c0a338d6f1873c6a8035aa"}
Jul 23 17:10:24.085448 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:24.085244    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af7qqbp" event={"ID":"b272f0fc-34de-4752-a5b2-a1591ca98bf0","Type":"ContainerStarted","Data":"71ec3b95183bc3cbdf9037d49de060976f49cb553d397c9eb5fa50743026b622"}
Jul 23 17:10:24.085724 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:24.085708    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 23 17:10:25.091276 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:25.091175    2577 generic.go:356] "Generic (PLEG): container finished" podID="b272f0fc-34de-4752-a5b2-a1591ca98bf0" containerID="d61a4d35be642f29ff848ff4b660e7d1b7bf66b8e70d44139198b3b751a871cd" exitCode=0
Jul 23 17:10:25.091276 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:25.091249    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af7qqbp" event={"ID":"b272f0fc-34de-4752-a5b2-a1591ca98bf0","Type":"ContainerDied","Data":"d61a4d35be642f29ff848ff4b660e7d1b7bf66b8e70d44139198b3b751a871cd"}
Jul 23 17:10:26.099098 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:26.099067    2577 generic.go:356] "Generic (PLEG): container finished" podID="b272f0fc-34de-4752-a5b2-a1591ca98bf0" containerID="abc97d2a51b0305486f03b1e2b6956f85542911c192e067760dbac150df7cf22" exitCode=0
Jul 23 17:10:26.099098 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:26.099105    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af7qqbp" event={"ID":"b272f0fc-34de-4752-a5b2-a1591ca98bf0","Type":"ContainerDied","Data":"abc97d2a51b0305486f03b1e2b6956f85542911c192e067760dbac150df7cf22"}
Jul 23 17:10:27.225704 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:27.225678    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af7qqbp"
Jul 23 17:10:27.270885 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:27.270852    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b272f0fc-34de-4752-a5b2-a1591ca98bf0-util\") pod \"b272f0fc-34de-4752-a5b2-a1591ca98bf0\" (UID: \"b272f0fc-34de-4752-a5b2-a1591ca98bf0\") "
Jul 23 17:10:27.271046 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:27.270890    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wwfhf\" (UniqueName: \"kubernetes.io/projected/b272f0fc-34de-4752-a5b2-a1591ca98bf0-kube-api-access-wwfhf\") pod \"b272f0fc-34de-4752-a5b2-a1591ca98bf0\" (UID: \"b272f0fc-34de-4752-a5b2-a1591ca98bf0\") "
Jul 23 17:10:27.271046 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:27.270917    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b272f0fc-34de-4752-a5b2-a1591ca98bf0-bundle\") pod \"b272f0fc-34de-4752-a5b2-a1591ca98bf0\" (UID: \"b272f0fc-34de-4752-a5b2-a1591ca98bf0\") "
Jul 23 17:10:27.271657 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:27.271623    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b272f0fc-34de-4752-a5b2-a1591ca98bf0-bundle" (OuterVolumeSpecName: "bundle") pod "b272f0fc-34de-4752-a5b2-a1591ca98bf0" (UID: "b272f0fc-34de-4752-a5b2-a1591ca98bf0"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 17:10:27.272851 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:27.272821    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b272f0fc-34de-4752-a5b2-a1591ca98bf0-kube-api-access-wwfhf" (OuterVolumeSpecName: "kube-api-access-wwfhf") pod "b272f0fc-34de-4752-a5b2-a1591ca98bf0" (UID: "b272f0fc-34de-4752-a5b2-a1591ca98bf0"). InnerVolumeSpecName "kube-api-access-wwfhf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 17:10:27.276736 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:27.276698    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b272f0fc-34de-4752-a5b2-a1591ca98bf0-util" (OuterVolumeSpecName: "util") pod "b272f0fc-34de-4752-a5b2-a1591ca98bf0" (UID: "b272f0fc-34de-4752-a5b2-a1591ca98bf0"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 17:10:27.372292 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:27.372194    2577 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b272f0fc-34de-4752-a5b2-a1591ca98bf0-util\") on node \"ip-10-0-129-216.ec2.internal\" DevicePath \"\""
Jul 23 17:10:27.372292 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:27.372221    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wwfhf\" (UniqueName: \"kubernetes.io/projected/b272f0fc-34de-4752-a5b2-a1591ca98bf0-kube-api-access-wwfhf\") on node \"ip-10-0-129-216.ec2.internal\" DevicePath \"\""
Jul 23 17:10:27.372292 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:27.372232    2577 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b272f0fc-34de-4752-a5b2-a1591ca98bf0-bundle\") on node \"ip-10-0-129-216.ec2.internal\" DevicePath \"\""
Jul 23 17:10:28.108960 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:28.108930    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af7qqbp"
Jul 23 17:10:28.108960 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:28.108943    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af7qqbp" event={"ID":"b272f0fc-34de-4752-a5b2-a1591ca98bf0","Type":"ContainerDied","Data":"71ec3b95183bc3cbdf9037d49de060976f49cb553d397c9eb5fa50743026b622"}
Jul 23 17:10:28.109162 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:28.108982    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="71ec3b95183bc3cbdf9037d49de060976f49cb553d397c9eb5fa50743026b622"
Jul 23 17:10:35.083328 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:35.083294    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:10:35.224068 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:35.224034    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-65fcfccbcc-2stkg"]
Jul 23 17:10:35.224351 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:35.224340    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b272f0fc-34de-4752-a5b2-a1591ca98bf0" containerName="util"
Jul 23 17:10:35.224401 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:35.224353    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="b272f0fc-34de-4752-a5b2-a1591ca98bf0" containerName="util"
Jul 23 17:10:35.224401 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:35.224364    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b272f0fc-34de-4752-a5b2-a1591ca98bf0" containerName="extract"
Jul 23 17:10:35.224401 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:35.224369    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="b272f0fc-34de-4752-a5b2-a1591ca98bf0" containerName="extract"
Jul 23 17:10:35.224401 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:35.224380    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b272f0fc-34de-4752-a5b2-a1591ca98bf0" containerName="pull"
Jul 23 17:10:35.224401 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:35.224385    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="b272f0fc-34de-4752-a5b2-a1591ca98bf0" containerName="pull"
Jul 23 17:10:35.224547 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:35.224446    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="b272f0fc-34de-4752-a5b2-a1591ca98bf0" containerName="extract"
Jul 23 17:10:35.226664 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:35.226649    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-65fcfccbcc-2stkg"
Jul 23 17:10:35.231054 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:35.231031    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Jul 23 17:10:35.232212 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:35.232193    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Jul 23 17:10:35.232441 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:35.232425    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Jul 23 17:10:35.232489 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:35.232448    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Jul 23 17:10:35.233169 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:35.233004    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-s8fpp\""
Jul 23 17:10:35.233169 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:35.233008    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Jul 23 17:10:35.241046 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:35.241021    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:10:35.241599 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:35.241577    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-65fcfccbcc-2stkg"]
Jul 23 17:10:35.336118 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:35.336043    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/dedd689a-b86a-4b58-81e0-93cbd069dcb9-manager-config\") pod \"lws-controller-manager-65fcfccbcc-2stkg\" (UID: \"dedd689a-b86a-4b58-81e0-93cbd069dcb9\") " pod="openshift-lws-operator/lws-controller-manager-65fcfccbcc-2stkg"
Jul 23 17:10:35.336118 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:35.336079    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/dedd689a-b86a-4b58-81e0-93cbd069dcb9-metrics-cert\") pod \"lws-controller-manager-65fcfccbcc-2stkg\" (UID: \"dedd689a-b86a-4b58-81e0-93cbd069dcb9\") " pod="openshift-lws-operator/lws-controller-manager-65fcfccbcc-2stkg"
Jul 23 17:10:35.336118 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:35.336120    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2b4h\" (UniqueName: \"kubernetes.io/projected/dedd689a-b86a-4b58-81e0-93cbd069dcb9-kube-api-access-r2b4h\") pod \"lws-controller-manager-65fcfccbcc-2stkg\" (UID: \"dedd689a-b86a-4b58-81e0-93cbd069dcb9\") " pod="openshift-lws-operator/lws-controller-manager-65fcfccbcc-2stkg"
Jul 23 17:10:35.336344 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:35.336170    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/dedd689a-b86a-4b58-81e0-93cbd069dcb9-cert\") pod \"lws-controller-manager-65fcfccbcc-2stkg\" (UID: \"dedd689a-b86a-4b58-81e0-93cbd069dcb9\") " pod="openshift-lws-operator/lws-controller-manager-65fcfccbcc-2stkg"
Jul 23 17:10:35.373405 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:35.373372    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:10:35.436731 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:35.436700    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2b4h\" (UniqueName: \"kubernetes.io/projected/dedd689a-b86a-4b58-81e0-93cbd069dcb9-kube-api-access-r2b4h\") pod \"lws-controller-manager-65fcfccbcc-2stkg\" (UID: \"dedd689a-b86a-4b58-81e0-93cbd069dcb9\") " pod="openshift-lws-operator/lws-controller-manager-65fcfccbcc-2stkg"
Jul 23 17:10:35.436884 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:35.436738    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/dedd689a-b86a-4b58-81e0-93cbd069dcb9-cert\") pod \"lws-controller-manager-65fcfccbcc-2stkg\" (UID: \"dedd689a-b86a-4b58-81e0-93cbd069dcb9\") " pod="openshift-lws-operator/lws-controller-manager-65fcfccbcc-2stkg"
Jul 23 17:10:35.436884 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:35.436770    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/dedd689a-b86a-4b58-81e0-93cbd069dcb9-manager-config\") pod \"lws-controller-manager-65fcfccbcc-2stkg\" (UID: \"dedd689a-b86a-4b58-81e0-93cbd069dcb9\") " pod="openshift-lws-operator/lws-controller-manager-65fcfccbcc-2stkg"
Jul 23 17:10:35.436983 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:35.436882    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/dedd689a-b86a-4b58-81e0-93cbd069dcb9-metrics-cert\") pod \"lws-controller-manager-65fcfccbcc-2stkg\" (UID: \"dedd689a-b86a-4b58-81e0-93cbd069dcb9\") " pod="openshift-lws-operator/lws-controller-manager-65fcfccbcc-2stkg"
Jul 23 17:10:35.437331 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:35.437314    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/dedd689a-b86a-4b58-81e0-93cbd069dcb9-manager-config\") pod \"lws-controller-manager-65fcfccbcc-2stkg\" (UID: \"dedd689a-b86a-4b58-81e0-93cbd069dcb9\") " pod="openshift-lws-operator/lws-controller-manager-65fcfccbcc-2stkg"
Jul 23 17:10:35.439191 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:35.439173    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/dedd689a-b86a-4b58-81e0-93cbd069dcb9-cert\") pod \"lws-controller-manager-65fcfccbcc-2stkg\" (UID: \"dedd689a-b86a-4b58-81e0-93cbd069dcb9\") " pod="openshift-lws-operator/lws-controller-manager-65fcfccbcc-2stkg"
Jul 23 17:10:35.439290 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:35.439203    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/dedd689a-b86a-4b58-81e0-93cbd069dcb9-metrics-cert\") pod \"lws-controller-manager-65fcfccbcc-2stkg\" (UID: \"dedd689a-b86a-4b58-81e0-93cbd069dcb9\") " pod="openshift-lws-operator/lws-controller-manager-65fcfccbcc-2stkg"
Jul 23 17:10:35.456906 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:35.456875    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2b4h\" (UniqueName: \"kubernetes.io/projected/dedd689a-b86a-4b58-81e0-93cbd069dcb9-kube-api-access-r2b4h\") pod \"lws-controller-manager-65fcfccbcc-2stkg\" (UID: \"dedd689a-b86a-4b58-81e0-93cbd069dcb9\") " pod="openshift-lws-operator/lws-controller-manager-65fcfccbcc-2stkg"
Jul 23 17:10:35.502292 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:35.502244    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:10:35.535537 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:35.535511    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-65fcfccbcc-2stkg"
Jul 23 17:10:35.698107 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:35.698084    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-65fcfccbcc-2stkg"]
Jul 23 17:10:35.700637 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:10:35.700606    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddedd689a_b86a_4b58_81e0_93cbd069dcb9.slice/crio-86eb5387476377bd742c55bae10c4cddc8430ada2a771703f5d16cf2e9eb25be WatchSource:0}: Error finding container 86eb5387476377bd742c55bae10c4cddc8430ada2a771703f5d16cf2e9eb25be: Status 404 returned error can't find the container with id 86eb5387476377bd742c55bae10c4cddc8430ada2a771703f5d16cf2e9eb25be
Jul 23 17:10:35.951955 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:35.951927    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:10:36.143130 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:36.143099    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-65fcfccbcc-2stkg" event={"ID":"dedd689a-b86a-4b58-81e0-93cbd069dcb9","Type":"ContainerStarted","Data":"86eb5387476377bd742c55bae10c4cddc8430ada2a771703f5d16cf2e9eb25be"}
Jul 23 17:10:36.184711 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:36.184684    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:10:36.283938 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:36.283855    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:10:36.411503 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:36.411467    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-f9b469455-874ln"]
Jul 23 17:10:36.414904 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:36.414888    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-f9b469455-874ln"
Jul 23 17:10:36.430970 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:36.430950    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Jul 23 17:10:36.432526 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:36.432511    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Jul 23 17:10:36.432964 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:36.432952    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Jul 23 17:10:36.434728 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:36.434703    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Jul 23 17:10:36.434833 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:36.434790    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-v7d57\""
Jul 23 17:10:36.455197 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:36.455173    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-f9b469455-874ln"]
Jul 23 17:10:36.545580 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:36.545503    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/317747fa-9885-43a3-a464-bbdb37ac9a6c-webhook-cert\") pod \"opendatahub-operator-controller-manager-f9b469455-874ln\" (UID: \"317747fa-9885-43a3-a464-bbdb37ac9a6c\") " pod="opendatahub/opendatahub-operator-controller-manager-f9b469455-874ln"
Jul 23 17:10:36.545580 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:36.545562    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/317747fa-9885-43a3-a464-bbdb37ac9a6c-manifests\") pod \"opendatahub-operator-controller-manager-f9b469455-874ln\" (UID: \"317747fa-9885-43a3-a464-bbdb37ac9a6c\") " pod="opendatahub/opendatahub-operator-controller-manager-f9b469455-874ln"
Jul 23 17:10:36.545770 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:36.545622    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/317747fa-9885-43a3-a464-bbdb37ac9a6c-apiservice-cert\") pod \"opendatahub-operator-controller-manager-f9b469455-874ln\" (UID: \"317747fa-9885-43a3-a464-bbdb37ac9a6c\") " pod="opendatahub/opendatahub-operator-controller-manager-f9b469455-874ln"
Jul 23 17:10:36.545770 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:36.545657    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/317747fa-9885-43a3-a464-bbdb37ac9a6c-tmp\") pod \"opendatahub-operator-controller-manager-f9b469455-874ln\" (UID: \"317747fa-9885-43a3-a464-bbdb37ac9a6c\") " pod="opendatahub/opendatahub-operator-controller-manager-f9b469455-874ln"
Jul 23 17:10:36.545770 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:36.545694    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqcfg\" (UniqueName: \"kubernetes.io/projected/317747fa-9885-43a3-a464-bbdb37ac9a6c-kube-api-access-wqcfg\") pod \"opendatahub-operator-controller-manager-f9b469455-874ln\" (UID: \"317747fa-9885-43a3-a464-bbdb37ac9a6c\") " pod="opendatahub/opendatahub-operator-controller-manager-f9b469455-874ln"
Jul 23 17:10:36.646965 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:36.646929    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/317747fa-9885-43a3-a464-bbdb37ac9a6c-tmp\") pod \"opendatahub-operator-controller-manager-f9b469455-874ln\" (UID: \"317747fa-9885-43a3-a464-bbdb37ac9a6c\") " pod="opendatahub/opendatahub-operator-controller-manager-f9b469455-874ln"
Jul 23 17:10:36.647200 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:36.646976    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqcfg\" (UniqueName: \"kubernetes.io/projected/317747fa-9885-43a3-a464-bbdb37ac9a6c-kube-api-access-wqcfg\") pod \"opendatahub-operator-controller-manager-f9b469455-874ln\" (UID: \"317747fa-9885-43a3-a464-bbdb37ac9a6c\") " pod="opendatahub/opendatahub-operator-controller-manager-f9b469455-874ln"
Jul 23 17:10:36.647200 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:36.647040    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/317747fa-9885-43a3-a464-bbdb37ac9a6c-webhook-cert\") pod \"opendatahub-operator-controller-manager-f9b469455-874ln\" (UID: \"317747fa-9885-43a3-a464-bbdb37ac9a6c\") " pod="opendatahub/opendatahub-operator-controller-manager-f9b469455-874ln"
Jul 23 17:10:36.647200 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:36.647080    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/317747fa-9885-43a3-a464-bbdb37ac9a6c-manifests\") pod \"opendatahub-operator-controller-manager-f9b469455-874ln\" (UID: \"317747fa-9885-43a3-a464-bbdb37ac9a6c\") " pod="opendatahub/opendatahub-operator-controller-manager-f9b469455-874ln"
Jul 23 17:10:36.647200 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:36.647151    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/317747fa-9885-43a3-a464-bbdb37ac9a6c-apiservice-cert\") pod \"opendatahub-operator-controller-manager-f9b469455-874ln\" (UID: \"317747fa-9885-43a3-a464-bbdb37ac9a6c\") " pod="opendatahub/opendatahub-operator-controller-manager-f9b469455-874ln"
Jul 23 17:10:36.647558 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:36.647528    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/317747fa-9885-43a3-a464-bbdb37ac9a6c-manifests\") pod \"opendatahub-operator-controller-manager-f9b469455-874ln\" (UID: \"317747fa-9885-43a3-a464-bbdb37ac9a6c\") " pod="opendatahub/opendatahub-operator-controller-manager-f9b469455-874ln"
Jul 23 17:10:36.650014 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:36.649992    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/317747fa-9885-43a3-a464-bbdb37ac9a6c-tmp\") pod \"opendatahub-operator-controller-manager-f9b469455-874ln\" (UID: \"317747fa-9885-43a3-a464-bbdb37ac9a6c\") " pod="opendatahub/opendatahub-operator-controller-manager-f9b469455-874ln"
Jul 23 17:10:36.650123 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:36.650063    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/317747fa-9885-43a3-a464-bbdb37ac9a6c-webhook-cert\") pod \"opendatahub-operator-controller-manager-f9b469455-874ln\" (UID: \"317747fa-9885-43a3-a464-bbdb37ac9a6c\") " pod="opendatahub/opendatahub-operator-controller-manager-f9b469455-874ln"
Jul 23 17:10:36.650123 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:36.650074    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/317747fa-9885-43a3-a464-bbdb37ac9a6c-apiservice-cert\") pod \"opendatahub-operator-controller-manager-f9b469455-874ln\" (UID: \"317747fa-9885-43a3-a464-bbdb37ac9a6c\") " pod="opendatahub/opendatahub-operator-controller-manager-f9b469455-874ln"
Jul 23 17:10:36.658500 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:36.658459    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqcfg\" (UniqueName: \"kubernetes.io/projected/317747fa-9885-43a3-a464-bbdb37ac9a6c-kube-api-access-wqcfg\") pod \"opendatahub-operator-controller-manager-f9b469455-874ln\" (UID: \"317747fa-9885-43a3-a464-bbdb37ac9a6c\") " pod="opendatahub/opendatahub-operator-controller-manager-f9b469455-874ln"
Jul 23 17:10:36.680885 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:36.680837    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:10:36.724407 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:36.724379    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-f9b469455-874ln"
Jul 23 17:10:36.778860 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:36.778831    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:10:36.883818 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:36.883784    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:10:36.884551 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:36.884526    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-f9b469455-874ln"]
Jul 23 17:10:36.888361 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:10:36.888338    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod317747fa_9885_43a3_a464_bbdb37ac9a6c.slice/crio-4a44f72ff752cbf0ddbb0cce1e991578802c51c5c2b51cd2c4d144993e4b2c59 WatchSource:0}: Error finding container 4a44f72ff752cbf0ddbb0cce1e991578802c51c5c2b51cd2c4d144993e4b2c59: Status 404 returned error can't find the container with id 4a44f72ff752cbf0ddbb0cce1e991578802c51c5c2b51cd2c4d144993e4b2c59
Jul 23 17:10:36.997625 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:36.997587    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:10:37.085795 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:37.085705    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:10:37.150188 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:37.150150    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-f9b469455-874ln" event={"ID":"317747fa-9885-43a3-a464-bbdb37ac9a6c","Type":"ContainerStarted","Data":"4a44f72ff752cbf0ddbb0cce1e991578802c51c5c2b51cd2c4d144993e4b2c59"}
Jul 23 17:10:37.217389 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:37.217351    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:10:38.156375 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:38.156338    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-65fcfccbcc-2stkg" event={"ID":"dedd689a-b86a-4b58-81e0-93cbd069dcb9","Type":"ContainerStarted","Data":"ac28d66e9ca92f023ce2753262f22405983bb4d21677aedcd61f40e573c95792"}
Jul 23 17:10:38.156798 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:38.156441    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-65fcfccbcc-2stkg"
Jul 23 17:10:38.185197 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:38.185141    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-65fcfccbcc-2stkg" podStartSLOduration=1.565968214 podStartE2EDuration="3.185121657s" totalImagesPullingTime="1.619153443s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 17:10:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 17:10:38.183625683 +0000 UTC m=+333.953911200" watchObservedRunningTime="2026-07-23 17:10:38.185121657 +0000 UTC m=+333.955407177"
Jul 23 17:10:40.168193 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:40.168148    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-f9b469455-874ln" event={"ID":"317747fa-9885-43a3-a464-bbdb37ac9a6c","Type":"ContainerStarted","Data":"dcffe455a84ac11887b6e592b0af0ee038f77bbeb5d8b24535b005b4083a6b8a"}
Jul 23 17:10:41.174311 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:41.174276    2577 generic.go:356] "Generic (PLEG): container finished" podID="317747fa-9885-43a3-a464-bbdb37ac9a6c" containerID="dcffe455a84ac11887b6e592b0af0ee038f77bbeb5d8b24535b005b4083a6b8a" exitCode=0
Jul 23 17:10:41.174708 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:41.174349    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-f9b469455-874ln" event={"ID":"317747fa-9885-43a3-a464-bbdb37ac9a6c","Type":"ContainerDied","Data":"dcffe455a84ac11887b6e592b0af0ee038f77bbeb5d8b24535b005b4083a6b8a"}
Jul 23 17:10:42.181894 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:42.181865    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-f9b469455-874ln" event={"ID":"317747fa-9885-43a3-a464-bbdb37ac9a6c","Type":"ContainerStarted","Data":"fcfc0af5666df1e221128c13db1cb950039c7d69ffc43035e16faf6e6ea6aa4e"}
Jul 23 17:10:42.182283 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:42.181975    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-f9b469455-874ln"
Jul 23 17:10:42.203578 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:42.203532    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-f9b469455-874ln" podStartSLOduration=1.19889987 podStartE2EDuration="6.203515452s" totalImagesPullingTime="3.993456694s" totalInitContainerRuntime="1.011158888s" isStatefulPod=false podCreationTimestamp="2026-07-23 17:10:36 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 17:10:42.202233895 +0000 UTC m=+337.972519411" watchObservedRunningTime="2026-07-23 17:10:42.203515452 +0000 UTC m=+337.973801035"
Jul 23 17:10:42.485584 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:42.485557    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:10:49.164353 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:49.164324    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-65fcfccbcc-2stkg"
Jul 23 17:10:53.189220 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:53.189185    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-f9b469455-874ln"
Jul 23 17:10:55.961519 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:55.961480    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:10:56.118982 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:56.118945    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:10:56.711796 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:56.711716    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:10:56.907016 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:56.906975    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:10:56.977216 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:56.977132    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-z4mjc"]
Jul 23 17:10:56.980592 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:56.980576    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-z4mjc"
Jul 23 17:10:57.003969 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:57.003944    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-5h4g4\""
Jul 23 17:10:57.004579 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:57.004563    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\""
Jul 23 17:10:57.038718 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:57.038692    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-z4mjc"]
Jul 23 17:10:57.118667 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:57.118637    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pd6cz\" (UniqueName: \"kubernetes.io/projected/aa92e815-3d31-4472-b7ef-ec1f76756bce-kube-api-access-pd6cz\") pod \"odh-model-controller-7998c67dc4-z4mjc\" (UID: \"aa92e815-3d31-4472-b7ef-ec1f76756bce\") " pod="opendatahub/odh-model-controller-7998c67dc4-z4mjc"
Jul 23 17:10:57.118826 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:57.118760    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/aa92e815-3d31-4472-b7ef-ec1f76756bce-cert\") pod \"odh-model-controller-7998c67dc4-z4mjc\" (UID: \"aa92e815-3d31-4472-b7ef-ec1f76756bce\") " pod="opendatahub/odh-model-controller-7998c67dc4-z4mjc"
Jul 23 17:10:57.146534 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:57.146486    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:10:57.219961 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:57.219925    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/aa92e815-3d31-4472-b7ef-ec1f76756bce-cert\") pod \"odh-model-controller-7998c67dc4-z4mjc\" (UID: \"aa92e815-3d31-4472-b7ef-ec1f76756bce\") " pod="opendatahub/odh-model-controller-7998c67dc4-z4mjc"
Jul 23 17:10:57.219961 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:57.219965    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pd6cz\" (UniqueName: \"kubernetes.io/projected/aa92e815-3d31-4472-b7ef-ec1f76756bce-kube-api-access-pd6cz\") pod \"odh-model-controller-7998c67dc4-z4mjc\" (UID: \"aa92e815-3d31-4472-b7ef-ec1f76756bce\") " pod="opendatahub/odh-model-controller-7998c67dc4-z4mjc"
Jul 23 17:10:57.220166 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:10:57.220072    2577 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found
Jul 23 17:10:57.220166 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:10:57.220142    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aa92e815-3d31-4472-b7ef-ec1f76756bce-cert podName:aa92e815-3d31-4472-b7ef-ec1f76756bce nodeName:}" failed. No retries permitted until 2026-07-23 17:10:57.720125684 +0000 UTC m=+353.490411181 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/aa92e815-3d31-4472-b7ef-ec1f76756bce-cert") pod "odh-model-controller-7998c67dc4-z4mjc" (UID: "aa92e815-3d31-4472-b7ef-ec1f76756bce") : secret "odh-model-controller-webhook-cert" not found
Jul 23 17:10:57.247178 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:57.247104    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pd6cz\" (UniqueName: \"kubernetes.io/projected/aa92e815-3d31-4472-b7ef-ec1f76756bce-kube-api-access-pd6cz\") pod \"odh-model-controller-7998c67dc4-z4mjc\" (UID: \"aa92e815-3d31-4472-b7ef-ec1f76756bce\") " pod="opendatahub/odh-model-controller-7998c67dc4-z4mjc"
Jul 23 17:10:57.303402 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:57.303370    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:10:57.442779 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:57.442741    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:10:57.712096 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:57.712056    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:10:57.725306 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:57.725270    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/aa92e815-3d31-4472-b7ef-ec1f76756bce-cert\") pod \"odh-model-controller-7998c67dc4-z4mjc\" (UID: \"aa92e815-3d31-4472-b7ef-ec1f76756bce\") " pod="opendatahub/odh-model-controller-7998c67dc4-z4mjc"
Jul 23 17:10:57.727815 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:57.727792    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/aa92e815-3d31-4472-b7ef-ec1f76756bce-cert\") pod \"odh-model-controller-7998c67dc4-z4mjc\" (UID: \"aa92e815-3d31-4472-b7ef-ec1f76756bce\") " pod="opendatahub/odh-model-controller-7998c67dc4-z4mjc"
Jul 23 17:10:57.890688 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:57.890655    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-z4mjc"
Jul 23 17:10:58.054359 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:58.054324    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-z4mjc"]
Jul 23 17:10:58.054717 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:10:58.054460    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa92e815_3d31_4472_b7ef_ec1f76756bce.slice/crio-71939e6147b90c025b992e380535356431b5952a0f294ab4aeea1b50ab5ce2c3 WatchSource:0}: Error finding container 71939e6147b90c025b992e380535356431b5952a0f294ab4aeea1b50ab5ce2c3: Status 404 returned error can't find the container with id 71939e6147b90c025b992e380535356431b5952a0f294ab4aeea1b50ab5ce2c3
Jul 23 17:10:58.258945 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:58.258851    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-z4mjc" event={"ID":"aa92e815-3d31-4472-b7ef-ec1f76756bce","Type":"ContainerStarted","Data":"71939e6147b90c025b992e380535356431b5952a0f294ab4aeea1b50ab5ce2c3"}
Jul 23 17:10:58.391785 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:58.391735    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:10:58.553395 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:58.553321    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:10:59.024993 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:59.024957    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:10:59.036070 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:59.035819    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-j75kh"]
Jul 23 17:10:59.039838 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:59.039811    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-j75kh"
Jul 23 17:10:59.046143 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:59.046119    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Jul 23 17:10:59.046143 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:59.046137    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 23 17:10:59.046420 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:59.046322    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Jul 23 17:10:59.046487 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:59.046462    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 23 17:10:59.046538 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:59.046505    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Jul 23 17:10:59.047075 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:59.046990    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 23 17:10:59.047336 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:59.047205    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-6lwtw\""
Jul 23 17:10:59.066090 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:59.066052    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-j75kh"]
Jul 23 17:10:59.138559 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:59.138517    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5834bca0-4f99-4181-8137-4d21b6d9c641-istio-token\") pod \"istiod-openshift-gateway-7574459656-j75kh\" (UID: \"5834bca0-4f99-4181-8137-4d21b6d9c641\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-j75kh"
Jul 23 17:10:59.138783 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:59.138575    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/5834bca0-4f99-4181-8137-4d21b6d9c641-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-j75kh\" (UID: \"5834bca0-4f99-4181-8137-4d21b6d9c641\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-j75kh"
Jul 23 17:10:59.138783 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:59.138616    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/5834bca0-4f99-4181-8137-4d21b6d9c641-cacerts\") pod \"istiod-openshift-gateway-7574459656-j75kh\" (UID: \"5834bca0-4f99-4181-8137-4d21b6d9c641\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-j75kh"
Jul 23 17:10:59.138783 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:59.138703    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ll65\" (UniqueName: \"kubernetes.io/projected/5834bca0-4f99-4181-8137-4d21b6d9c641-kube-api-access-2ll65\") pod \"istiod-openshift-gateway-7574459656-j75kh\" (UID: \"5834bca0-4f99-4181-8137-4d21b6d9c641\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-j75kh"
Jul 23 17:10:59.138783 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:59.138754    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5834bca0-4f99-4181-8137-4d21b6d9c641-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-j75kh\" (UID: \"5834bca0-4f99-4181-8137-4d21b6d9c641\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-j75kh"
Jul 23 17:10:59.139154 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:59.138784    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/5834bca0-4f99-4181-8137-4d21b6d9c641-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-j75kh\" (UID: \"5834bca0-4f99-4181-8137-4d21b6d9c641\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-j75kh"
Jul 23 17:10:59.139154 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:59.138829    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/5834bca0-4f99-4181-8137-4d21b6d9c641-local-certs\") pod \"istiod-openshift-gateway-7574459656-j75kh\" (UID: \"5834bca0-4f99-4181-8137-4d21b6d9c641\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-j75kh"
Jul 23 17:10:59.219891 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:59.219827    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:10:59.239909 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:59.239868    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5834bca0-4f99-4181-8137-4d21b6d9c641-istio-token\") pod \"istiod-openshift-gateway-7574459656-j75kh\" (UID: \"5834bca0-4f99-4181-8137-4d21b6d9c641\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-j75kh"
Jul 23 17:10:59.240090 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:59.239916    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/5834bca0-4f99-4181-8137-4d21b6d9c641-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-j75kh\" (UID: \"5834bca0-4f99-4181-8137-4d21b6d9c641\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-j75kh"
Jul 23 17:10:59.240333 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:59.240309    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/5834bca0-4f99-4181-8137-4d21b6d9c641-cacerts\") pod \"istiod-openshift-gateway-7574459656-j75kh\" (UID: \"5834bca0-4f99-4181-8137-4d21b6d9c641\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-j75kh"
Jul 23 17:10:59.240443 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:59.240390    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ll65\" (UniqueName: \"kubernetes.io/projected/5834bca0-4f99-4181-8137-4d21b6d9c641-kube-api-access-2ll65\") pod \"istiod-openshift-gateway-7574459656-j75kh\" (UID: \"5834bca0-4f99-4181-8137-4d21b6d9c641\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-j75kh"
Jul 23 17:10:59.240443 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:59.240424    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5834bca0-4f99-4181-8137-4d21b6d9c641-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-j75kh\" (UID: \"5834bca0-4f99-4181-8137-4d21b6d9c641\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-j75kh"
Jul 23 17:10:59.240557 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:59.240451    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/5834bca0-4f99-4181-8137-4d21b6d9c641-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-j75kh\" (UID: \"5834bca0-4f99-4181-8137-4d21b6d9c641\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-j75kh"
Jul 23 17:10:59.240557 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:59.240498    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/5834bca0-4f99-4181-8137-4d21b6d9c641-local-certs\") pod \"istiod-openshift-gateway-7574459656-j75kh\" (UID: \"5834bca0-4f99-4181-8137-4d21b6d9c641\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-j75kh"
Jul 23 17:10:59.241732 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:59.241696    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/5834bca0-4f99-4181-8137-4d21b6d9c641-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-j75kh\" (UID: \"5834bca0-4f99-4181-8137-4d21b6d9c641\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-j75kh"
Jul 23 17:10:59.245132 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:59.245090    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5834bca0-4f99-4181-8137-4d21b6d9c641-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-j75kh\" (UID: \"5834bca0-4f99-4181-8137-4d21b6d9c641\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-j75kh"
Jul 23 17:10:59.246166 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:59.246125    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/5834bca0-4f99-4181-8137-4d21b6d9c641-local-certs\") pod \"istiod-openshift-gateway-7574459656-j75kh\" (UID: \"5834bca0-4f99-4181-8137-4d21b6d9c641\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-j75kh"
Jul 23 17:10:59.246868 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:59.246683    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/5834bca0-4f99-4181-8137-4d21b6d9c641-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-j75kh\" (UID: \"5834bca0-4f99-4181-8137-4d21b6d9c641\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-j75kh"
Jul 23 17:10:59.246868 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:59.246830    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/5834bca0-4f99-4181-8137-4d21b6d9c641-cacerts\") pod \"istiod-openshift-gateway-7574459656-j75kh\" (UID: \"5834bca0-4f99-4181-8137-4d21b6d9c641\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-j75kh"
Jul 23 17:10:59.278072 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:59.277999    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ll65\" (UniqueName: \"kubernetes.io/projected/5834bca0-4f99-4181-8137-4d21b6d9c641-kube-api-access-2ll65\") pod \"istiod-openshift-gateway-7574459656-j75kh\" (UID: \"5834bca0-4f99-4181-8137-4d21b6d9c641\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-j75kh"
Jul 23 17:10:59.283548 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:59.283521    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5834bca0-4f99-4181-8137-4d21b6d9c641-istio-token\") pod \"istiod-openshift-gateway-7574459656-j75kh\" (UID: \"5834bca0-4f99-4181-8137-4d21b6d9c641\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-j75kh"
Jul 23 17:10:59.354017 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:59.353982    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-j75kh"
Jul 23 17:10:59.542798 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:10:59.542668    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-j75kh"]
Jul 23 17:10:59.547768 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:10:59.547735    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5834bca0_4f99_4181_8137_4d21b6d9c641.slice/crio-650cfc76482e567b5b0efb014fc6bec557b9954ec4c42031cf18f792e705986e WatchSource:0}: Error finding container 650cfc76482e567b5b0efb014fc6bec557b9954ec4c42031cf18f792e705986e: Status 404 returned error can't find the container with id 650cfc76482e567b5b0efb014fc6bec557b9954ec4c42031cf18f792e705986e
Jul 23 17:11:00.285540 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:11:00.285501    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-j75kh" event={"ID":"5834bca0-4f99-4181-8137-4d21b6d9c641","Type":"ContainerStarted","Data":"650cfc76482e567b5b0efb014fc6bec557b9954ec4c42031cf18f792e705986e"}
Jul 23 17:11:00.914800 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:11:00.914752    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:11:01.293612 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:11:01.293574    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-z4mjc" event={"ID":"aa92e815-3d31-4472-b7ef-ec1f76756bce","Type":"ContainerStarted","Data":"138b04f5c3b186b8dbed964c19c6142aac36d00ddb91434cced4bc6405762522"}
Jul 23 17:11:01.294270 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:11:01.293756    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-z4mjc"
Jul 23 17:11:01.314482 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:11:01.314423    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-z4mjc" podStartSLOduration=2.276451497 podStartE2EDuration="5.314403601s" totalImagesPullingTime="3.037952104s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 17:10:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 17:11:01.312890794 +0000 UTC m=+357.083176313" watchObservedRunningTime="2026-07-23 17:11:01.314403601 +0000 UTC m=+357.084689122"
Jul 23 17:11:03.053841 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:11:03.053806    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 23 17:11:03.054090 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:11:03.053874    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 23 17:11:03.305415 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:11:03.305329    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-j75kh" event={"ID":"5834bca0-4f99-4181-8137-4d21b6d9c641","Type":"ContainerStarted","Data":"98670e05c37d08ea53668206a0b86a71264a0ec58fc9278eb29b31041ff42068"}
Jul 23 17:11:03.305560 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:11:03.305484    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-j75kh"
Jul 23 17:11:03.332608 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:11:03.332556    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7574459656-j75kh" podStartSLOduration=1.8293952660000001 podStartE2EDuration="5.332538719s" totalImagesPullingTime="3.503143453s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 17:10:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 17:11:03.329864289 +0000 UTC m=+359.100149808" watchObservedRunningTime="2026-07-23 17:11:03.332538719 +0000 UTC m=+359.102824237"
Jul 23 17:11:03.753188 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:11:03.753146    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:11:03.849791 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:11:03.849758    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:11:04.121551 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:11:04.121473    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:11:04.311943 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:11:04.311916    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-j75kh"
Jul 23 17:11:04.357653 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:11:04.357617    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:11:04.618991 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:11:04.618958    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:11:08.423001 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:11:08.422965    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:11:08.578139 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:11:08.578109    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-6svpm"]
Jul 23 17:11:08.582893 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:11:08.582855    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:11:08.587496 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:11:08.587475    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-6svpm"
Jul 23 17:11:08.592321 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:11:08.592300    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\""
Jul 23 17:11:08.592558 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:11:08.592537    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-qzwx8\""
Jul 23 17:11:08.618793 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:11:08.618766    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-6svpm"]
Jul 23 17:11:08.729996 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:11:08.729962    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqxkk\" (UniqueName: \"kubernetes.io/projected/4e7eff36-6da5-43e5-be32-820f601c91c7-kube-api-access-sqxkk\") pod \"llmisvc-controller-manager-777c6bd7d5-6svpm\" (UID: \"4e7eff36-6da5-43e5-be32-820f601c91c7\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-6svpm"
Jul 23 17:11:08.730154 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:11:08.730018    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4e7eff36-6da5-43e5-be32-820f601c91c7-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-6svpm\" (UID: \"4e7eff36-6da5-43e5-be32-820f601c91c7\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-6svpm"
Jul 23 17:11:08.831012 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:11:08.830977    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqxkk\" (UniqueName: \"kubernetes.io/projected/4e7eff36-6da5-43e5-be32-820f601c91c7-kube-api-access-sqxkk\") pod \"llmisvc-controller-manager-777c6bd7d5-6svpm\" (UID: \"4e7eff36-6da5-43e5-be32-820f601c91c7\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-6svpm"
Jul 23 17:11:08.831180 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:11:08.831039    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4e7eff36-6da5-43e5-be32-820f601c91c7-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-6svpm\" (UID: \"4e7eff36-6da5-43e5-be32-820f601c91c7\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-6svpm"
Jul 23 17:11:08.831231 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:11:08.831180    2577 secret.go:189] Couldn't get secret opendatahub/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found
Jul 23 17:11:08.831320 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:11:08.831249    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4e7eff36-6da5-43e5-be32-820f601c91c7-cert podName:4e7eff36-6da5-43e5-be32-820f601c91c7 nodeName:}" failed. No retries permitted until 2026-07-23 17:11:09.331231959 +0000 UTC m=+365.101517460 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/4e7eff36-6da5-43e5-be32-820f601c91c7-cert") pod "llmisvc-controller-manager-777c6bd7d5-6svpm" (UID: "4e7eff36-6da5-43e5-be32-820f601c91c7") : secret "llmisvc-webhook-server-cert" not found
Jul 23 17:11:08.852929 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:11:08.852903    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqxkk\" (UniqueName: \"kubernetes.io/projected/4e7eff36-6da5-43e5-be32-820f601c91c7-kube-api-access-sqxkk\") pod \"llmisvc-controller-manager-777c6bd7d5-6svpm\" (UID: \"4e7eff36-6da5-43e5-be32-820f601c91c7\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-6svpm"
Jul 23 17:11:08.946376 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:11:08.946344    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:11:09.138277 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:11:09.138180    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:11:09.288243 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:11:09.288210    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:11:09.336533 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:11:09.336502    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4e7eff36-6da5-43e5-be32-820f601c91c7-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-6svpm\" (UID: \"4e7eff36-6da5-43e5-be32-820f601c91c7\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-6svpm"
Jul 23 17:11:09.339019 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:11:09.338997    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4e7eff36-6da5-43e5-be32-820f601c91c7-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-6svpm\" (UID: \"4e7eff36-6da5-43e5-be32-820f601c91c7\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-6svpm"
Jul 23 17:11:09.497387 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:11:09.497349    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-6svpm"
Jul 23 17:11:09.657538 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:11:09.657506    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod4e7eff36_6da5_43e5_be32_820f601c91c7.slice/crio-67cd7d79e72735b601594d1b12dfaf0c0e2ebc7e1c95ed74f2a322c79a1caca5 WatchSource:0}: Error finding container 67cd7d79e72735b601594d1b12dfaf0c0e2ebc7e1c95ed74f2a322c79a1caca5: Status 404 returned error can't find the container with id 67cd7d79e72735b601594d1b12dfaf0c0e2ebc7e1c95ed74f2a322c79a1caca5
Jul 23 17:11:09.660067 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:11:09.660046    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-6svpm"]
Jul 23 17:11:10.272190 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:11:10.272146    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-j75kh"]
Jul 23 17:11:10.272634 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:11:10.272459    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7574459656-j75kh" podUID="5834bca0-4f99-4181-8137-4d21b6d9c641" containerName="discovery" containerID="cri-o://98670e05c37d08ea53668206a0b86a71264a0ec58fc9278eb29b31041ff42068" gracePeriod=30
Jul 23 17:11:10.310854 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:11:10.310824    2577 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7574459656-j75kh container/discovery namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.26:8080/ready\": dial tcp 10.133.0.26:8080: connect: connection refused" start-of-body=
Jul 23 17:11:10.310971 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:11:10.310888    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7574459656-j75kh" podUID="5834bca0-4f99-4181-8137-4d21b6d9c641" containerName="discovery" probeResult="failure" output="Get \"http://10.133.0.26:8080/ready\": dial tcp 10.133.0.26:8080: connect: connection refused"
Jul 23 17:11:10.341636 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:11:10.341608    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-6svpm" event={"ID":"4e7eff36-6da5-43e5-be32-820f601c91c7","Type":"ContainerStarted","Data":"67cd7d79e72735b601594d1b12dfaf0c0e2ebc7e1c95ed74f2a322c79a1caca5"}
Jul 23 17:11:10.545160 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:11:10.545136    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-j75kh"
Jul 23 17:11:10.646091 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:11:10.646060    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5834bca0-4f99-4181-8137-4d21b6d9c641-istio-token\") pod \"5834bca0-4f99-4181-8137-4d21b6d9c641\" (UID: \"5834bca0-4f99-4181-8137-4d21b6d9c641\") "
Jul 23 17:11:10.646299 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:11:10.646138    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/5834bca0-4f99-4181-8137-4d21b6d9c641-cacerts\") pod \"5834bca0-4f99-4181-8137-4d21b6d9c641\" (UID: \"5834bca0-4f99-4181-8137-4d21b6d9c641\") "
Jul 23 17:11:10.646299 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:11:10.646164    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5834bca0-4f99-4181-8137-4d21b6d9c641-istio-kubeconfig\") pod \"5834bca0-4f99-4181-8137-4d21b6d9c641\" (UID: \"5834bca0-4f99-4181-8137-4d21b6d9c641\") "
Jul 23 17:11:10.646299 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:11:10.646191    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/5834bca0-4f99-4181-8137-4d21b6d9c641-istio-csr-ca-configmap\") pod \"5834bca0-4f99-4181-8137-4d21b6d9c641\" (UID: \"5834bca0-4f99-4181-8137-4d21b6d9c641\") "
Jul 23 17:11:10.646299 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:11:10.646226    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/5834bca0-4f99-4181-8137-4d21b6d9c641-istio-csr-dns-cert\") pod \"5834bca0-4f99-4181-8137-4d21b6d9c641\" (UID: \"5834bca0-4f99-4181-8137-4d21b6d9c641\") "
Jul 23 17:11:10.646511 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:11:10.646326    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2ll65\" (UniqueName: \"kubernetes.io/projected/5834bca0-4f99-4181-8137-4d21b6d9c641-kube-api-access-2ll65\") pod \"5834bca0-4f99-4181-8137-4d21b6d9c641\" (UID: \"5834bca0-4f99-4181-8137-4d21b6d9c641\") "
Jul 23 17:11:10.646511 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:11:10.646366    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/5834bca0-4f99-4181-8137-4d21b6d9c641-local-certs\") pod \"5834bca0-4f99-4181-8137-4d21b6d9c641\" (UID: \"5834bca0-4f99-4181-8137-4d21b6d9c641\") "
Jul 23 17:11:10.646976 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:11:10.646936    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5834bca0-4f99-4181-8137-4d21b6d9c641-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "5834bca0-4f99-4181-8137-4d21b6d9c641" (UID: "5834bca0-4f99-4181-8137-4d21b6d9c641"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 17:11:10.649422 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:11:10.649088    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5834bca0-4f99-4181-8137-4d21b6d9c641-cacerts" (OuterVolumeSpecName: "cacerts") pod "5834bca0-4f99-4181-8137-4d21b6d9c641" (UID: "5834bca0-4f99-4181-8137-4d21b6d9c641"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 17:11:10.649422 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:11:10.649203    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5834bca0-4f99-4181-8137-4d21b6d9c641-istio-token" (OuterVolumeSpecName: "istio-token") pod "5834bca0-4f99-4181-8137-4d21b6d9c641" (UID: "5834bca0-4f99-4181-8137-4d21b6d9c641"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 17:11:10.649597 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:11:10.649477    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5834bca0-4f99-4181-8137-4d21b6d9c641-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "5834bca0-4f99-4181-8137-4d21b6d9c641" (UID: "5834bca0-4f99-4181-8137-4d21b6d9c641"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 17:11:10.649597 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:11:10.649548    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5834bca0-4f99-4181-8137-4d21b6d9c641-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "5834bca0-4f99-4181-8137-4d21b6d9c641" (UID: "5834bca0-4f99-4181-8137-4d21b6d9c641"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 17:11:10.649856 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:11:10.649831    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5834bca0-4f99-4181-8137-4d21b6d9c641-local-certs" (OuterVolumeSpecName: "local-certs") pod "5834bca0-4f99-4181-8137-4d21b6d9c641" (UID: "5834bca0-4f99-4181-8137-4d21b6d9c641"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 17:11:10.650083 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:11:10.650061    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5834bca0-4f99-4181-8137-4d21b6d9c641-kube-api-access-2ll65" (OuterVolumeSpecName: "kube-api-access-2ll65") pod "5834bca0-4f99-4181-8137-4d21b6d9c641" (UID: "5834bca0-4f99-4181-8137-4d21b6d9c641"). InnerVolumeSpecName "kube-api-access-2ll65". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 17:11:10.747714 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:11:10.747676    2577 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/5834bca0-4f99-4181-8137-4d21b6d9c641-istio-csr-dns-cert\") on node \"ip-10-0-129-216.ec2.internal\" DevicePath \"\""
Jul 23 17:11:10.747714 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:11:10.747712    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2ll65\" (UniqueName: \"kubernetes.io/projected/5834bca0-4f99-4181-8137-4d21b6d9c641-kube-api-access-2ll65\") on node \"ip-10-0-129-216.ec2.internal\" DevicePath \"\""
Jul 23 17:11:10.747914 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:11:10.747727    2577 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/5834bca0-4f99-4181-8137-4d21b6d9c641-local-certs\") on node \"ip-10-0-129-216.ec2.internal\" DevicePath \"\""
Jul 23 17:11:10.747914 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:11:10.747739    2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5834bca0-4f99-4181-8137-4d21b6d9c641-istio-token\") on node \"ip-10-0-129-216.ec2.internal\" DevicePath \"\""
Jul 23 17:11:10.747914 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:11:10.747751    2577 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/5834bca0-4f99-4181-8137-4d21b6d9c641-cacerts\") on node \"ip-10-0-129-216.ec2.internal\" DevicePath \"\""
Jul 23 17:11:10.747914 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:11:10.747762    2577 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5834bca0-4f99-4181-8137-4d21b6d9c641-istio-kubeconfig\") on node \"ip-10-0-129-216.ec2.internal\" DevicePath \"\""
Jul 23 17:11:10.747914 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:11:10.747772    2577 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/5834bca0-4f99-4181-8137-4d21b6d9c641-istio-csr-ca-configmap\") on node \"ip-10-0-129-216.ec2.internal\" DevicePath \"\""
Jul 23 17:11:11.348240 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:11:11.348202    2577 generic.go:356] "Generic (PLEG): container finished" podID="5834bca0-4f99-4181-8137-4d21b6d9c641" containerID="98670e05c37d08ea53668206a0b86a71264a0ec58fc9278eb29b31041ff42068" exitCode=0
Jul 23 17:11:11.348437 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:11:11.348249    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-j75kh"
Jul 23 17:11:11.348437 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:11:11.348290    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-j75kh" event={"ID":"5834bca0-4f99-4181-8137-4d21b6d9c641","Type":"ContainerDied","Data":"98670e05c37d08ea53668206a0b86a71264a0ec58fc9278eb29b31041ff42068"}
Jul 23 17:11:11.348437 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:11:11.348347    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-j75kh" event={"ID":"5834bca0-4f99-4181-8137-4d21b6d9c641","Type":"ContainerDied","Data":"650cfc76482e567b5b0efb014fc6bec557b9954ec4c42031cf18f792e705986e"}
Jul 23 17:11:11.348437 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:11:11.348372    2577 scope.go:117] "RemoveContainer" containerID="98670e05c37d08ea53668206a0b86a71264a0ec58fc9278eb29b31041ff42068"
Jul 23 17:11:11.358382 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:11:11.358361    2577 scope.go:117] "RemoveContainer" containerID="98670e05c37d08ea53668206a0b86a71264a0ec58fc9278eb29b31041ff42068"
Jul 23 17:11:11.358683 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:11:11.358665    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"98670e05c37d08ea53668206a0b86a71264a0ec58fc9278eb29b31041ff42068\": container with ID starting with 98670e05c37d08ea53668206a0b86a71264a0ec58fc9278eb29b31041ff42068 not found: ID does not exist" containerID="98670e05c37d08ea53668206a0b86a71264a0ec58fc9278eb29b31041ff42068"
Jul 23 17:11:11.358772 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:11:11.358691    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98670e05c37d08ea53668206a0b86a71264a0ec58fc9278eb29b31041ff42068"} err="failed to get container status \"98670e05c37d08ea53668206a0b86a71264a0ec58fc9278eb29b31041ff42068\": rpc error: code = NotFound desc = could not find container \"98670e05c37d08ea53668206a0b86a71264a0ec58fc9278eb29b31041ff42068\": container with ID starting with 98670e05c37d08ea53668206a0b86a71264a0ec58fc9278eb29b31041ff42068 not found: ID does not exist"
Jul 23 17:11:11.372532 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:11:11.372496    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-j75kh"]
Jul 23 17:11:11.377101 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:11:11.377076    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-j75kh"]
Jul 23 17:11:12.301818 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:11:12.301788    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-z4mjc"
Jul 23 17:11:12.767822 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:11:12.767785    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5834bca0-4f99-4181-8137-4d21b6d9c641" path="/var/lib/kubelet/pods/5834bca0-4f99-4181-8137-4d21b6d9c641/volumes"
Jul 23 17:11:13.363765 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:11:13.363663    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-6svpm" event={"ID":"4e7eff36-6da5-43e5-be32-820f601c91c7","Type":"ContainerStarted","Data":"332a363a84beae041fc2ead1482bf4b750dc288348749f9889ab3bcb9ac5b4cf"}
Jul 23 17:11:13.364219 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:11:13.363830    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-6svpm"
Jul 23 17:11:13.395202 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:11:13.395147    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-6svpm" podStartSLOduration=2.06939024 podStartE2EDuration="5.395132423s" totalImagesPullingTime="3.325742183s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 17:11:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 17:11:13.38783961 +0000 UTC m=+369.158125130" watchObservedRunningTime="2026-07-23 17:11:13.395132423 +0000 UTC m=+369.165417972"
Jul 23 17:11:44.370349 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:11:44.370314    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-6svpm"
Jul 23 17:11:57.764675 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:11:57.764644    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:11:57.862083 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:11:57.862052    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:11:57.894612 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:11:57.894585    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-8jqnb"]
Jul 23 17:11:57.894908 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:11:57.894894    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5834bca0-4f99-4181-8137-4d21b6d9c641" containerName="discovery"
Jul 23 17:11:57.894957 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:11:57.894909    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="5834bca0-4f99-4181-8137-4d21b6d9c641" containerName="discovery"
Jul 23 17:11:57.894993 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:11:57.894979    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="5834bca0-4f99-4181-8137-4d21b6d9c641" containerName="discovery"
Jul 23 17:11:57.897345 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:11:57.897328    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-8jqnb"
Jul 23 17:11:57.900217 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:11:57.900198    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 23 17:11:57.900905 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:11:57.900884    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 23 17:11:57.901008 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:11:57.900923    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-sh94q\""
Jul 23 17:11:57.910730 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:11:57.910700    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-8jqnb"]
Jul 23 17:11:57.971446 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:11:57.971413    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:11:58.042785 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:11:58.042702    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzpzh\" (UniqueName: \"kubernetes.io/projected/3d718b30-00c4-479d-b35b-f8d159da476e-kube-api-access-bzpzh\") pod \"kuadrant-operator-catalog-8jqnb\" (UID: \"3d718b30-00c4-479d-b35b-f8d159da476e\") " pod="kuadrant-system/kuadrant-operator-catalog-8jqnb"
Jul 23 17:11:58.143867 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:11:58.143831    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzpzh\" (UniqueName: \"kubernetes.io/projected/3d718b30-00c4-479d-b35b-f8d159da476e-kube-api-access-bzpzh\") pod \"kuadrant-operator-catalog-8jqnb\" (UID: \"3d718b30-00c4-479d-b35b-f8d159da476e\") " pod="kuadrant-system/kuadrant-operator-catalog-8jqnb"
Jul 23 17:11:58.152031 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:11:58.152001    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzpzh\" (UniqueName: \"kubernetes.io/projected/3d718b30-00c4-479d-b35b-f8d159da476e-kube-api-access-bzpzh\") pod \"kuadrant-operator-catalog-8jqnb\" (UID: \"3d718b30-00c4-479d-b35b-f8d159da476e\") " pod="kuadrant-system/kuadrant-operator-catalog-8jqnb"
Jul 23 17:11:58.207296 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:11:58.207245    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-8jqnb"
Jul 23 17:11:58.252070 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:11:58.252034    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-8jqnb"]
Jul 23 17:11:58.337602 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:11:58.337582    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-8jqnb"]
Jul 23 17:11:58.339631 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:11:58.339602    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d718b30_00c4_479d_b35b_f8d159da476e.slice/crio-8eee50db1aae13ce797bb0985f1193b88c222bde28c2edd176a0639f0dd0c0d7 WatchSource:0}: Error finding container 8eee50db1aae13ce797bb0985f1193b88c222bde28c2edd176a0639f0dd0c0d7: Status 404 returned error can't find the container with id 8eee50db1aae13ce797bb0985f1193b88c222bde28c2edd176a0639f0dd0c0d7
Jul 23 17:11:58.483615 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:11:58.483586    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-bd4jn"]
Jul 23 17:11:58.486809 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:11:58.486794    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-bd4jn"
Jul 23 17:11:58.497298 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:11:58.497247    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-bd4jn"]
Jul 23 17:11:58.578808 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:11:58.578775    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-8jqnb" event={"ID":"3d718b30-00c4-479d-b35b-f8d159da476e","Type":"ContainerStarted","Data":"8eee50db1aae13ce797bb0985f1193b88c222bde28c2edd176a0639f0dd0c0d7"}
Jul 23 17:11:58.648474 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:11:58.648380    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mfr99\" (UniqueName: \"kubernetes.io/projected/d572c939-0f51-4f05-950c-fe8ce11eacd3-kube-api-access-mfr99\") pod \"kuadrant-operator-catalog-bd4jn\" (UID: \"d572c939-0f51-4f05-950c-fe8ce11eacd3\") " pod="kuadrant-system/kuadrant-operator-catalog-bd4jn"
Jul 23 17:11:58.690646 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:11:58.690612    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:11:58.749167 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:11:58.749123    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mfr99\" (UniqueName: \"kubernetes.io/projected/d572c939-0f51-4f05-950c-fe8ce11eacd3-kube-api-access-mfr99\") pod \"kuadrant-operator-catalog-bd4jn\" (UID: \"d572c939-0f51-4f05-950c-fe8ce11eacd3\") " pod="kuadrant-system/kuadrant-operator-catalog-bd4jn"
Jul 23 17:11:58.757676 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:11:58.757640    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mfr99\" (UniqueName: \"kubernetes.io/projected/d572c939-0f51-4f05-950c-fe8ce11eacd3-kube-api-access-mfr99\") pod \"kuadrant-operator-catalog-bd4jn\" (UID: \"d572c939-0f51-4f05-950c-fe8ce11eacd3\") " pod="kuadrant-system/kuadrant-operator-catalog-bd4jn"
Jul 23 17:11:58.796577 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:11:58.796546    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-bd4jn"
Jul 23 17:11:58.891889 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:11:58.891856    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:11:58.926897 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:11:58.926869    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-bd4jn"]
Jul 23 17:11:58.928876 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:11:58.928851    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd572c939_0f51_4f05_950c_fe8ce11eacd3.slice/crio-86fc44382a2608ff866cc4bdee220ec2870e577f3f0833dcd8f03749b87544c0 WatchSource:0}: Error finding container 86fc44382a2608ff866cc4bdee220ec2870e577f3f0833dcd8f03749b87544c0: Status 404 returned error can't find the container with id 86fc44382a2608ff866cc4bdee220ec2870e577f3f0833dcd8f03749b87544c0
Jul 23 17:11:59.586314 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:11:59.586273    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-bd4jn" event={"ID":"d572c939-0f51-4f05-950c-fe8ce11eacd3","Type":"ContainerStarted","Data":"86fc44382a2608ff866cc4bdee220ec2870e577f3f0833dcd8f03749b87544c0"}
Jul 23 17:12:01.597156 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:01.597123    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-bd4jn" event={"ID":"d572c939-0f51-4f05-950c-fe8ce11eacd3","Type":"ContainerStarted","Data":"7f2040dd04b3490f368577bd28b6c6d32d76556398caa8258964703a9af16930"}
Jul 23 17:12:01.598663 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:01.598639    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-8jqnb" event={"ID":"3d718b30-00c4-479d-b35b-f8d159da476e","Type":"ContainerStarted","Data":"f2ff13516e13528541efd936a7f536a396cced3b9a3ce8d60cbc9bdcb0705eac"}
Jul 23 17:12:01.598778 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:01.598658    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-8jqnb" podUID="3d718b30-00c4-479d-b35b-f8d159da476e" containerName="registry-server" containerID="cri-o://f2ff13516e13528541efd936a7f536a396cced3b9a3ce8d60cbc9bdcb0705eac" gracePeriod=2
Jul 23 17:12:01.614327 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:01.614284    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-bd4jn" podStartSLOduration=1.598953806 podStartE2EDuration="3.614269504s" totalImagesPullingTime="2.015315698s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 17:11:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 17:12:01.613485429 +0000 UTC m=+417.383770962" watchObservedRunningTime="2026-07-23 17:12:01.614269504 +0000 UTC m=+417.384555014"
Jul 23 17:12:01.628718 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:01.628676    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-8jqnb" podStartSLOduration=2.019842137 podStartE2EDuration="4.628661374s" totalImagesPullingTime="2.608819237s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 17:11:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 17:12:01.627965409 +0000 UTC m=+417.398250928" watchObservedRunningTime="2026-07-23 17:12:01.628661374 +0000 UTC m=+417.398946892"
Jul 23 17:12:01.843815 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:01.843789    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-8jqnb"
Jul 23 17:12:01.977862 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:01.977835    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bzpzh\" (UniqueName: \"kubernetes.io/projected/3d718b30-00c4-479d-b35b-f8d159da476e-kube-api-access-bzpzh\") pod \"3d718b30-00c4-479d-b35b-f8d159da476e\" (UID: \"3d718b30-00c4-479d-b35b-f8d159da476e\") "
Jul 23 17:12:01.979982 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:01.979944    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d718b30-00c4-479d-b35b-f8d159da476e-kube-api-access-bzpzh" (OuterVolumeSpecName: "kube-api-access-bzpzh") pod "3d718b30-00c4-479d-b35b-f8d159da476e" (UID: "3d718b30-00c4-479d-b35b-f8d159da476e"). InnerVolumeSpecName "kube-api-access-bzpzh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 17:12:02.078805 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:02.078765    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bzpzh\" (UniqueName: \"kubernetes.io/projected/3d718b30-00c4-479d-b35b-f8d159da476e-kube-api-access-bzpzh\") on node \"ip-10-0-129-216.ec2.internal\" DevicePath \"\""
Jul 23 17:12:02.604206 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:02.604173    2577 generic.go:356] "Generic (PLEG): container finished" podID="3d718b30-00c4-479d-b35b-f8d159da476e" containerID="f2ff13516e13528541efd936a7f536a396cced3b9a3ce8d60cbc9bdcb0705eac" exitCode=0
Jul 23 17:12:02.604668 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:02.604219    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-8jqnb"
Jul 23 17:12:02.604668 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:02.604277    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-8jqnb" event={"ID":"3d718b30-00c4-479d-b35b-f8d159da476e","Type":"ContainerDied","Data":"f2ff13516e13528541efd936a7f536a396cced3b9a3ce8d60cbc9bdcb0705eac"}
Jul 23 17:12:02.604668 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:02.604338    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-8jqnb" event={"ID":"3d718b30-00c4-479d-b35b-f8d159da476e","Type":"ContainerDied","Data":"8eee50db1aae13ce797bb0985f1193b88c222bde28c2edd176a0639f0dd0c0d7"}
Jul 23 17:12:02.604668 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:02.604370    2577 scope.go:117] "RemoveContainer" containerID="f2ff13516e13528541efd936a7f536a396cced3b9a3ce8d60cbc9bdcb0705eac"
Jul 23 17:12:02.613318 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:02.613303    2577 scope.go:117] "RemoveContainer" containerID="f2ff13516e13528541efd936a7f536a396cced3b9a3ce8d60cbc9bdcb0705eac"
Jul 23 17:12:02.613539 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:12:02.613521    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2ff13516e13528541efd936a7f536a396cced3b9a3ce8d60cbc9bdcb0705eac\": container with ID starting with f2ff13516e13528541efd936a7f536a396cced3b9a3ce8d60cbc9bdcb0705eac not found: ID does not exist" containerID="f2ff13516e13528541efd936a7f536a396cced3b9a3ce8d60cbc9bdcb0705eac"
Jul 23 17:12:02.613588 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:02.613547    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2ff13516e13528541efd936a7f536a396cced3b9a3ce8d60cbc9bdcb0705eac"} err="failed to get container status \"f2ff13516e13528541efd936a7f536a396cced3b9a3ce8d60cbc9bdcb0705eac\": rpc error: code = NotFound desc = could not find container \"f2ff13516e13528541efd936a7f536a396cced3b9a3ce8d60cbc9bdcb0705eac\": container with ID starting with f2ff13516e13528541efd936a7f536a396cced3b9a3ce8d60cbc9bdcb0705eac not found: ID does not exist"
Jul 23 17:12:02.625471 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:02.625450    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-8jqnb"]
Jul 23 17:12:02.629154 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:02.629135    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-8jqnb"]
Jul 23 17:12:02.765411 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:02.765382    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3d718b30-00c4-479d-b35b-f8d159da476e" path="/var/lib/kubelet/pods/3d718b30-00c4-479d-b35b-f8d159da476e/volumes"
Jul 23 17:12:04.978115 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:04.978074    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:12:08.797094 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:08.797055    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-bd4jn"
Jul 23 17:12:08.797635 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:08.797118    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-bd4jn"
Jul 23 17:12:08.819358 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:08.819333    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-bd4jn"
Jul 23 17:12:09.666276 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:09.666218    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-bd4jn"
Jul 23 17:12:17.009132 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:17.009092    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xpmjs"]
Jul 23 17:12:17.009738 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:17.009494    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d718b30-00c4-479d-b35b-f8d159da476e" containerName="registry-server"
Jul 23 17:12:17.009738 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:17.009507    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d718b30-00c4-479d-b35b-f8d159da476e" containerName="registry-server"
Jul 23 17:12:17.009738 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:17.009593    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d718b30-00c4-479d-b35b-f8d159da476e" containerName="registry-server"
Jul 23 17:12:17.013087 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:17.013064    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xpmjs"
Jul 23 17:12:17.016732 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:17.016712    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-qbxzr\""
Jul 23 17:12:17.029298 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:17.029244    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xpmjs"]
Jul 23 17:12:17.100028 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:17.099983    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xf7w2\" (UniqueName: \"kubernetes.io/projected/62a7ca3d-ef60-422c-bef3-627044060c5f-kube-api-access-xf7w2\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xpmjs\" (UID: \"62a7ca3d-ef60-422c-bef3-627044060c5f\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xpmjs"
Jul 23 17:12:17.100211 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:17.100047    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/62a7ca3d-ef60-422c-bef3-627044060c5f-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xpmjs\" (UID: \"62a7ca3d-ef60-422c-bef3-627044060c5f\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xpmjs"
Jul 23 17:12:17.100211 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:17.100080    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/62a7ca3d-ef60-422c-bef3-627044060c5f-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xpmjs\" (UID: \"62a7ca3d-ef60-422c-bef3-627044060c5f\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xpmjs"
Jul 23 17:12:17.116629 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:17.116600    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jqkqj"]
Jul 23 17:12:17.119335 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:17.119318    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jqkqj"
Jul 23 17:12:17.129860 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:17.129836    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jqkqj"]
Jul 23 17:12:17.200716 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:17.200674    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gh782\" (UniqueName: \"kubernetes.io/projected/f0f53fe8-060d-4ddc-94a6-c361cc5d8688-kube-api-access-gh782\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jqkqj\" (UID: \"f0f53fe8-060d-4ddc-94a6-c361cc5d8688\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jqkqj"
Jul 23 17:12:17.200975 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:17.200731    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xf7w2\" (UniqueName: \"kubernetes.io/projected/62a7ca3d-ef60-422c-bef3-627044060c5f-kube-api-access-xf7w2\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xpmjs\" (UID: \"62a7ca3d-ef60-422c-bef3-627044060c5f\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xpmjs"
Jul 23 17:12:17.200975 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:17.200783    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f0f53fe8-060d-4ddc-94a6-c361cc5d8688-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jqkqj\" (UID: \"f0f53fe8-060d-4ddc-94a6-c361cc5d8688\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jqkqj"
Jul 23 17:12:17.200975 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:17.200835    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/62a7ca3d-ef60-422c-bef3-627044060c5f-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xpmjs\" (UID: \"62a7ca3d-ef60-422c-bef3-627044060c5f\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xpmjs"
Jul 23 17:12:17.200975 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:17.200870    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/62a7ca3d-ef60-422c-bef3-627044060c5f-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xpmjs\" (UID: \"62a7ca3d-ef60-422c-bef3-627044060c5f\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xpmjs"
Jul 23 17:12:17.200975 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:17.200901    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f0f53fe8-060d-4ddc-94a6-c361cc5d8688-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jqkqj\" (UID: \"f0f53fe8-060d-4ddc-94a6-c361cc5d8688\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jqkqj"
Jul 23 17:12:17.201224 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:17.201203    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/62a7ca3d-ef60-422c-bef3-627044060c5f-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xpmjs\" (UID: \"62a7ca3d-ef60-422c-bef3-627044060c5f\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xpmjs"
Jul 23 17:12:17.201291 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:17.201230    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/62a7ca3d-ef60-422c-bef3-627044060c5f-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xpmjs\" (UID: \"62a7ca3d-ef60-422c-bef3-627044060c5f\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xpmjs"
Jul 23 17:12:17.211111 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:17.211085    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xf7w2\" (UniqueName: \"kubernetes.io/projected/62a7ca3d-ef60-422c-bef3-627044060c5f-kube-api-access-xf7w2\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xpmjs\" (UID: \"62a7ca3d-ef60-422c-bef3-627044060c5f\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xpmjs"
Jul 23 17:12:17.301362 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:17.301236    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f0f53fe8-060d-4ddc-94a6-c361cc5d8688-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jqkqj\" (UID: \"f0f53fe8-060d-4ddc-94a6-c361cc5d8688\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jqkqj"
Jul 23 17:12:17.301362 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:17.301345    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gh782\" (UniqueName: \"kubernetes.io/projected/f0f53fe8-060d-4ddc-94a6-c361cc5d8688-kube-api-access-gh782\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jqkqj\" (UID: \"f0f53fe8-060d-4ddc-94a6-c361cc5d8688\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jqkqj"
Jul 23 17:12:17.301597 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:17.301397    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f0f53fe8-060d-4ddc-94a6-c361cc5d8688-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jqkqj\" (UID: \"f0f53fe8-060d-4ddc-94a6-c361cc5d8688\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jqkqj"
Jul 23 17:12:17.301659 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:17.301605    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f0f53fe8-060d-4ddc-94a6-c361cc5d8688-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jqkqj\" (UID: \"f0f53fe8-060d-4ddc-94a6-c361cc5d8688\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jqkqj"
Jul 23 17:12:17.301747 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:17.301728    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f0f53fe8-060d-4ddc-94a6-c361cc5d8688-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jqkqj\" (UID: \"f0f53fe8-060d-4ddc-94a6-c361cc5d8688\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jqkqj"
Jul 23 17:12:17.310116 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:17.310092    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gh782\" (UniqueName: \"kubernetes.io/projected/f0f53fe8-060d-4ddc-94a6-c361cc5d8688-kube-api-access-gh782\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jqkqj\" (UID: \"f0f53fe8-060d-4ddc-94a6-c361cc5d8688\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jqkqj"
Jul 23 17:12:17.326919 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:17.326894    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xpmjs"
Jul 23 17:12:17.422922 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:17.422883    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xchhq"]
Jul 23 17:12:17.427817 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:17.427796    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xchhq"
Jul 23 17:12:17.427986 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:17.427962    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jqkqj"
Jul 23 17:12:17.435872 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:17.435841    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xchhq"]
Jul 23 17:12:17.468177 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:17.468144    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xpmjs"]
Jul 23 17:12:17.471105 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:12:17.471076    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62a7ca3d_ef60_422c_bef3_627044060c5f.slice/crio-1dbdf18a85251dcbad38396fbc9bd01283b2b0dfb341aecdcc71ae7f963296c1 WatchSource:0}: Error finding container 1dbdf18a85251dcbad38396fbc9bd01283b2b0dfb341aecdcc71ae7f963296c1: Status 404 returned error can't find the container with id 1dbdf18a85251dcbad38396fbc9bd01283b2b0dfb341aecdcc71ae7f963296c1
Jul 23 17:12:17.502519 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:17.502480    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/283546e1-df7c-49a0-a72a-2318766fc257-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xchhq\" (UID: \"283546e1-df7c-49a0-a72a-2318766fc257\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xchhq"
Jul 23 17:12:17.502650 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:17.502527    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/283546e1-df7c-49a0-a72a-2318766fc257-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xchhq\" (UID: \"283546e1-df7c-49a0-a72a-2318766fc257\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xchhq"
Jul 23 17:12:17.502650 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:17.502556    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bbhgx\" (UniqueName: \"kubernetes.io/projected/283546e1-df7c-49a0-a72a-2318766fc257-kube-api-access-bbhgx\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xchhq\" (UID: \"283546e1-df7c-49a0-a72a-2318766fc257\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xchhq"
Jul 23 17:12:17.566948 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:17.566923    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jqkqj"]
Jul 23 17:12:17.568642 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:12:17.568614    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0f53fe8_060d_4ddc_94a6_c361cc5d8688.slice/crio-99cb1180be20a5c2a4dfe220726052273f6e3835811e34dd40f47633fb281a4d WatchSource:0}: Error finding container 99cb1180be20a5c2a4dfe220726052273f6e3835811e34dd40f47633fb281a4d: Status 404 returned error can't find the container with id 99cb1180be20a5c2a4dfe220726052273f6e3835811e34dd40f47633fb281a4d
Jul 23 17:12:17.603472 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:17.603435    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/283546e1-df7c-49a0-a72a-2318766fc257-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xchhq\" (UID: \"283546e1-df7c-49a0-a72a-2318766fc257\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xchhq"
Jul 23 17:12:17.603762 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:17.603480    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/283546e1-df7c-49a0-a72a-2318766fc257-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xchhq\" (UID: \"283546e1-df7c-49a0-a72a-2318766fc257\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xchhq"
Jul 23 17:12:17.603762 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:17.603515    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bbhgx\" (UniqueName: \"kubernetes.io/projected/283546e1-df7c-49a0-a72a-2318766fc257-kube-api-access-bbhgx\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xchhq\" (UID: \"283546e1-df7c-49a0-a72a-2318766fc257\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xchhq"
Jul 23 17:12:17.604013 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:17.603858    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/283546e1-df7c-49a0-a72a-2318766fc257-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xchhq\" (UID: \"283546e1-df7c-49a0-a72a-2318766fc257\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xchhq"
Jul 23 17:12:17.604013 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:17.603967    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/283546e1-df7c-49a0-a72a-2318766fc257-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xchhq\" (UID: \"283546e1-df7c-49a0-a72a-2318766fc257\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xchhq"
Jul 23 17:12:17.611898 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:17.611879    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bbhgx\" (UniqueName: \"kubernetes.io/projected/283546e1-df7c-49a0-a72a-2318766fc257-kube-api-access-bbhgx\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xchhq\" (UID: \"283546e1-df7c-49a0-a72a-2318766fc257\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xchhq"
Jul 23 17:12:17.684280 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:17.684228    2577 generic.go:356] "Generic (PLEG): container finished" podID="62a7ca3d-ef60-422c-bef3-627044060c5f" containerID="1f3e12df75a5a79af551315cc8f9666c6e702a253bff85fb58fa81dae56ed0f0" exitCode=0
Jul 23 17:12:17.684422 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:17.684290    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xpmjs" event={"ID":"62a7ca3d-ef60-422c-bef3-627044060c5f","Type":"ContainerDied","Data":"1f3e12df75a5a79af551315cc8f9666c6e702a253bff85fb58fa81dae56ed0f0"}
Jul 23 17:12:17.684422 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:17.684347    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xpmjs" event={"ID":"62a7ca3d-ef60-422c-bef3-627044060c5f","Type":"ContainerStarted","Data":"1dbdf18a85251dcbad38396fbc9bd01283b2b0dfb341aecdcc71ae7f963296c1"}
Jul 23 17:12:17.686342 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:17.686318    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jqkqj" event={"ID":"f0f53fe8-060d-4ddc-94a6-c361cc5d8688","Type":"ContainerStarted","Data":"d622ab891ff0d209d97dacf1132f57daa58b0e0186f73bc19f34063b4e5e6377"}
Jul 23 17:12:17.686450 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:17.686362    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jqkqj" event={"ID":"f0f53fe8-060d-4ddc-94a6-c361cc5d8688","Type":"ContainerStarted","Data":"99cb1180be20a5c2a4dfe220726052273f6e3835811e34dd40f47633fb281a4d"}
Jul 23 17:12:17.749195 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:17.749162    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xchhq"
Jul 23 17:12:17.821088 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:17.821025    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759zx982"]
Jul 23 17:12:17.824215 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:17.824198    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759zx982"
Jul 23 17:12:17.839549 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:17.839523    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759zx982"]
Jul 23 17:12:17.887144 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:17.887117    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xchhq"]
Jul 23 17:12:17.888607 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:12:17.888584    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod283546e1_df7c_49a0_a72a_2318766fc257.slice/crio-c6704f31e71366d636d0b81140a2980dea0f2e09f2160190c172174772518bba WatchSource:0}: Error finding container c6704f31e71366d636d0b81140a2980dea0f2e09f2160190c172174772518bba: Status 404 returned error can't find the container with id c6704f31e71366d636d0b81140a2980dea0f2e09f2160190c172174772518bba
Jul 23 17:12:17.905746 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:17.905722    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2pwxw\" (UniqueName: \"kubernetes.io/projected/67c1a8e8-ea9e-4d80-b4e8-cecbb603a745-kube-api-access-2pwxw\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759zx982\" (UID: \"67c1a8e8-ea9e-4d80-b4e8-cecbb603a745\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759zx982"
Jul 23 17:12:17.905831 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:17.905777    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/67c1a8e8-ea9e-4d80-b4e8-cecbb603a745-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759zx982\" (UID: \"67c1a8e8-ea9e-4d80-b4e8-cecbb603a745\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759zx982"
Jul 23 17:12:17.905831 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:17.905812    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/67c1a8e8-ea9e-4d80-b4e8-cecbb603a745-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759zx982\" (UID: \"67c1a8e8-ea9e-4d80-b4e8-cecbb603a745\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759zx982"
Jul 23 17:12:18.006452 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:18.006421    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/67c1a8e8-ea9e-4d80-b4e8-cecbb603a745-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759zx982\" (UID: \"67c1a8e8-ea9e-4d80-b4e8-cecbb603a745\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759zx982"
Jul 23 17:12:18.006626 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:18.006484    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/67c1a8e8-ea9e-4d80-b4e8-cecbb603a745-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759zx982\" (UID: \"67c1a8e8-ea9e-4d80-b4e8-cecbb603a745\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759zx982"
Jul 23 17:12:18.006626 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:18.006537    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2pwxw\" (UniqueName: \"kubernetes.io/projected/67c1a8e8-ea9e-4d80-b4e8-cecbb603a745-kube-api-access-2pwxw\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759zx982\" (UID: \"67c1a8e8-ea9e-4d80-b4e8-cecbb603a745\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759zx982"
Jul 23 17:12:18.006891 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:18.006868    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/67c1a8e8-ea9e-4d80-b4e8-cecbb603a745-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759zx982\" (UID: \"67c1a8e8-ea9e-4d80-b4e8-cecbb603a745\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759zx982"
Jul 23 17:12:18.006969 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:18.006871    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/67c1a8e8-ea9e-4d80-b4e8-cecbb603a745-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759zx982\" (UID: \"67c1a8e8-ea9e-4d80-b4e8-cecbb603a745\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759zx982"
Jul 23 17:12:18.016236 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:18.016210    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2pwxw\" (UniqueName: \"kubernetes.io/projected/67c1a8e8-ea9e-4d80-b4e8-cecbb603a745-kube-api-access-2pwxw\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759zx982\" (UID: \"67c1a8e8-ea9e-4d80-b4e8-cecbb603a745\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759zx982"
Jul 23 17:12:18.135120 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:18.135042    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759zx982"
Jul 23 17:12:18.266153 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:18.266130    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759zx982"]
Jul 23 17:12:18.268098 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:12:18.268067    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67c1a8e8_ea9e_4d80_b4e8_cecbb603a745.slice/crio-e894fc072a4fc55c5de461f6152e583b68dbdb0ee4dd60c4514c0b7d19b8a922 WatchSource:0}: Error finding container e894fc072a4fc55c5de461f6152e583b68dbdb0ee4dd60c4514c0b7d19b8a922: Status 404 returned error can't find the container with id e894fc072a4fc55c5de461f6152e583b68dbdb0ee4dd60c4514c0b7d19b8a922
Jul 23 17:12:18.693109 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:18.693073    2577 generic.go:356] "Generic (PLEG): container finished" podID="67c1a8e8-ea9e-4d80-b4e8-cecbb603a745" containerID="2620b92e1939cdcab23529d86afbda10e3f0e492e8be51671362dd37a554dd15" exitCode=0
Jul 23 17:12:18.693336 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:18.693155    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759zx982" event={"ID":"67c1a8e8-ea9e-4d80-b4e8-cecbb603a745","Type":"ContainerDied","Data":"2620b92e1939cdcab23529d86afbda10e3f0e492e8be51671362dd37a554dd15"}
Jul 23 17:12:18.693336 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:18.693205    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759zx982" event={"ID":"67c1a8e8-ea9e-4d80-b4e8-cecbb603a745","Type":"ContainerStarted","Data":"e894fc072a4fc55c5de461f6152e583b68dbdb0ee4dd60c4514c0b7d19b8a922"}
Jul 23 17:12:18.694993 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:18.694970    2577 generic.go:356] "Generic (PLEG): container finished" podID="62a7ca3d-ef60-422c-bef3-627044060c5f" containerID="f0cf5f54da6fd1e12f2a6dadf84cca60d0dc4184330c9d4140f0d995f7eee60d" exitCode=0
Jul 23 17:12:18.695085 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:18.695059    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xpmjs" event={"ID":"62a7ca3d-ef60-422c-bef3-627044060c5f","Type":"ContainerDied","Data":"f0cf5f54da6fd1e12f2a6dadf84cca60d0dc4184330c9d4140f0d995f7eee60d"}
Jul 23 17:12:18.696595 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:18.696531    2577 generic.go:356] "Generic (PLEG): container finished" podID="f0f53fe8-060d-4ddc-94a6-c361cc5d8688" containerID="d622ab891ff0d209d97dacf1132f57daa58b0e0186f73bc19f34063b4e5e6377" exitCode=0
Jul 23 17:12:18.696595 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:18.696588    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jqkqj" event={"ID":"f0f53fe8-060d-4ddc-94a6-c361cc5d8688","Type":"ContainerDied","Data":"d622ab891ff0d209d97dacf1132f57daa58b0e0186f73bc19f34063b4e5e6377"}
Jul 23 17:12:18.698232 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:18.698210    2577 generic.go:356] "Generic (PLEG): container finished" podID="283546e1-df7c-49a0-a72a-2318766fc257" containerID="c4d62cf42a30135f720f7564eafff50863a694d22f91826961232a8a67721668" exitCode=0
Jul 23 17:12:18.698353 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:18.698279    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xchhq" event={"ID":"283546e1-df7c-49a0-a72a-2318766fc257","Type":"ContainerDied","Data":"c4d62cf42a30135f720f7564eafff50863a694d22f91826961232a8a67721668"}
Jul 23 17:12:18.698353 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:18.698307    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xchhq" event={"ID":"283546e1-df7c-49a0-a72a-2318766fc257","Type":"ContainerStarted","Data":"c6704f31e71366d636d0b81140a2980dea0f2e09f2160190c172174772518bba"}
Jul 23 17:12:19.704839 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:19.704800    2577 generic.go:356] "Generic (PLEG): container finished" podID="62a7ca3d-ef60-422c-bef3-627044060c5f" containerID="2e23ffcb0e79d5d166ca05506c69a0bd6275cb13fc11a0dcc1b5b3509920acf5" exitCode=0
Jul 23 17:12:19.705222 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:19.704872    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xpmjs" event={"ID":"62a7ca3d-ef60-422c-bef3-627044060c5f","Type":"ContainerDied","Data":"2e23ffcb0e79d5d166ca05506c69a0bd6275cb13fc11a0dcc1b5b3509920acf5"}
Jul 23 17:12:19.706515 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:19.706494    2577 generic.go:356] "Generic (PLEG): container finished" podID="283546e1-df7c-49a0-a72a-2318766fc257" containerID="ce89497a1449cde29e7f88fae6f6ef659161112797490d10b8ef9650645adc4a" exitCode=0
Jul 23 17:12:19.706618 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:19.706525    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xchhq" event={"ID":"283546e1-df7c-49a0-a72a-2318766fc257","Type":"ContainerDied","Data":"ce89497a1449cde29e7f88fae6f6ef659161112797490d10b8ef9650645adc4a"}
Jul 23 17:12:20.713598 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:20.713562    2577 generic.go:356] "Generic (PLEG): container finished" podID="f0f53fe8-060d-4ddc-94a6-c361cc5d8688" containerID="e31684f17e1637f743bcd629422facad7ad7dd9350e476850ff81803d80d6148" exitCode=0
Jul 23 17:12:20.714003 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:20.713632    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jqkqj" event={"ID":"f0f53fe8-060d-4ddc-94a6-c361cc5d8688","Type":"ContainerDied","Data":"e31684f17e1637f743bcd629422facad7ad7dd9350e476850ff81803d80d6148"}
Jul 23 17:12:20.715786 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:20.715762    2577 generic.go:356] "Generic (PLEG): container finished" podID="283546e1-df7c-49a0-a72a-2318766fc257" containerID="ab5674ae208b7a54a3d81e6f665857752067c87296b40ae3ca2ac382514848f2" exitCode=0
Jul 23 17:12:20.715863 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:20.715848    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xchhq" event={"ID":"283546e1-df7c-49a0-a72a-2318766fc257","Type":"ContainerDied","Data":"ab5674ae208b7a54a3d81e6f665857752067c87296b40ae3ca2ac382514848f2"}
Jul 23 17:12:20.717588 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:20.717563    2577 generic.go:356] "Generic (PLEG): container finished" podID="67c1a8e8-ea9e-4d80-b4e8-cecbb603a745" containerID="5111332fd7503ba89deda369d19684f7645c01a748717fa4d0c383049e45d1a0" exitCode=0
Jul 23 17:12:20.717682 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:20.717605    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759zx982" event={"ID":"67c1a8e8-ea9e-4d80-b4e8-cecbb603a745","Type":"ContainerDied","Data":"5111332fd7503ba89deda369d19684f7645c01a748717fa4d0c383049e45d1a0"}
Jul 23 17:12:20.970138 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:20.970079    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xpmjs"
Jul 23 17:12:21.034418 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:21.034387    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/62a7ca3d-ef60-422c-bef3-627044060c5f-util\") pod \"62a7ca3d-ef60-422c-bef3-627044060c5f\" (UID: \"62a7ca3d-ef60-422c-bef3-627044060c5f\") "
Jul 23 17:12:21.034562 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:21.034427    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xf7w2\" (UniqueName: \"kubernetes.io/projected/62a7ca3d-ef60-422c-bef3-627044060c5f-kube-api-access-xf7w2\") pod \"62a7ca3d-ef60-422c-bef3-627044060c5f\" (UID: \"62a7ca3d-ef60-422c-bef3-627044060c5f\") "
Jul 23 17:12:21.034562 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:21.034452    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/62a7ca3d-ef60-422c-bef3-627044060c5f-bundle\") pod \"62a7ca3d-ef60-422c-bef3-627044060c5f\" (UID: \"62a7ca3d-ef60-422c-bef3-627044060c5f\") "
Jul 23 17:12:21.034926 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:21.034902    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62a7ca3d-ef60-422c-bef3-627044060c5f-bundle" (OuterVolumeSpecName: "bundle") pod "62a7ca3d-ef60-422c-bef3-627044060c5f" (UID: "62a7ca3d-ef60-422c-bef3-627044060c5f"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 17:12:21.036483 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:21.036459    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62a7ca3d-ef60-422c-bef3-627044060c5f-kube-api-access-xf7w2" (OuterVolumeSpecName: "kube-api-access-xf7w2") pod "62a7ca3d-ef60-422c-bef3-627044060c5f" (UID: "62a7ca3d-ef60-422c-bef3-627044060c5f"). InnerVolumeSpecName "kube-api-access-xf7w2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 17:12:21.039693 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:21.039659    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62a7ca3d-ef60-422c-bef3-627044060c5f-util" (OuterVolumeSpecName: "util") pod "62a7ca3d-ef60-422c-bef3-627044060c5f" (UID: "62a7ca3d-ef60-422c-bef3-627044060c5f"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 17:12:21.135235 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:21.135201    2577 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/62a7ca3d-ef60-422c-bef3-627044060c5f-util\") on node \"ip-10-0-129-216.ec2.internal\" DevicePath \"\""
Jul 23 17:12:21.135235 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:21.135229    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xf7w2\" (UniqueName: \"kubernetes.io/projected/62a7ca3d-ef60-422c-bef3-627044060c5f-kube-api-access-xf7w2\") on node \"ip-10-0-129-216.ec2.internal\" DevicePath \"\""
Jul 23 17:12:21.135235 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:21.135240    2577 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/62a7ca3d-ef60-422c-bef3-627044060c5f-bundle\") on node \"ip-10-0-129-216.ec2.internal\" DevicePath \"\""
Jul 23 17:12:21.725205 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:21.725170    2577 generic.go:356] "Generic (PLEG): container finished" podID="67c1a8e8-ea9e-4d80-b4e8-cecbb603a745" containerID="36fd325517eedc026a9609b12e84808eb7facea25602044d48150e3eb5580639" exitCode=0
Jul 23 17:12:21.725609 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:21.725279    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759zx982" event={"ID":"67c1a8e8-ea9e-4d80-b4e8-cecbb603a745","Type":"ContainerDied","Data":"36fd325517eedc026a9609b12e84808eb7facea25602044d48150e3eb5580639"}
Jul 23 17:12:21.727088 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:21.727060    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xpmjs" event={"ID":"62a7ca3d-ef60-422c-bef3-627044060c5f","Type":"ContainerDied","Data":"1dbdf18a85251dcbad38396fbc9bd01283b2b0dfb341aecdcc71ae7f963296c1"}
Jul 23 17:12:21.727088 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:21.727075    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xpmjs"
Jul 23 17:12:21.727267 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:21.727102    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1dbdf18a85251dcbad38396fbc9bd01283b2b0dfb341aecdcc71ae7f963296c1"
Jul 23 17:12:21.729045 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:21.729024    2577 generic.go:356] "Generic (PLEG): container finished" podID="f0f53fe8-060d-4ddc-94a6-c361cc5d8688" containerID="6d10414367d28e5941c52651698371ef8e204067e8513db0ef3d505dcce12c33" exitCode=0
Jul 23 17:12:21.729149 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:21.729088    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jqkqj" event={"ID":"f0f53fe8-060d-4ddc-94a6-c361cc5d8688","Type":"ContainerDied","Data":"6d10414367d28e5941c52651698371ef8e204067e8513db0ef3d505dcce12c33"}
Jul 23 17:12:21.860218 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:21.860196    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xchhq"
Jul 23 17:12:21.943242 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:21.943209    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/283546e1-df7c-49a0-a72a-2318766fc257-util\") pod \"283546e1-df7c-49a0-a72a-2318766fc257\" (UID: \"283546e1-df7c-49a0-a72a-2318766fc257\") "
Jul 23 17:12:21.943242 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:21.943247    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bbhgx\" (UniqueName: \"kubernetes.io/projected/283546e1-df7c-49a0-a72a-2318766fc257-kube-api-access-bbhgx\") pod \"283546e1-df7c-49a0-a72a-2318766fc257\" (UID: \"283546e1-df7c-49a0-a72a-2318766fc257\") "
Jul 23 17:12:21.943450 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:21.943303    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/283546e1-df7c-49a0-a72a-2318766fc257-bundle\") pod \"283546e1-df7c-49a0-a72a-2318766fc257\" (UID: \"283546e1-df7c-49a0-a72a-2318766fc257\") "
Jul 23 17:12:21.943951 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:21.943914    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/283546e1-df7c-49a0-a72a-2318766fc257-bundle" (OuterVolumeSpecName: "bundle") pod "283546e1-df7c-49a0-a72a-2318766fc257" (UID: "283546e1-df7c-49a0-a72a-2318766fc257"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 17:12:21.945361 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:21.945335    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/283546e1-df7c-49a0-a72a-2318766fc257-kube-api-access-bbhgx" (OuterVolumeSpecName: "kube-api-access-bbhgx") pod "283546e1-df7c-49a0-a72a-2318766fc257" (UID: "283546e1-df7c-49a0-a72a-2318766fc257"). InnerVolumeSpecName "kube-api-access-bbhgx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 17:12:21.948685 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:21.948650    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/283546e1-df7c-49a0-a72a-2318766fc257-util" (OuterVolumeSpecName: "util") pod "283546e1-df7c-49a0-a72a-2318766fc257" (UID: "283546e1-df7c-49a0-a72a-2318766fc257"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 17:12:22.044172 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:22.044088    2577 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/283546e1-df7c-49a0-a72a-2318766fc257-util\") on node \"ip-10-0-129-216.ec2.internal\" DevicePath \"\""
Jul 23 17:12:22.044172 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:22.044120    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bbhgx\" (UniqueName: \"kubernetes.io/projected/283546e1-df7c-49a0-a72a-2318766fc257-kube-api-access-bbhgx\") on node \"ip-10-0-129-216.ec2.internal\" DevicePath \"\""
Jul 23 17:12:22.044172 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:22.044131    2577 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/283546e1-df7c-49a0-a72a-2318766fc257-bundle\") on node \"ip-10-0-129-216.ec2.internal\" DevicePath \"\""
Jul 23 17:12:22.735853 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:22.735819    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xchhq"
Jul 23 17:12:22.735853 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:22.735846    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xchhq" event={"ID":"283546e1-df7c-49a0-a72a-2318766fc257","Type":"ContainerDied","Data":"c6704f31e71366d636d0b81140a2980dea0f2e09f2160190c172174772518bba"}
Jul 23 17:12:22.736309 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:22.735919    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c6704f31e71366d636d0b81140a2980dea0f2e09f2160190c172174772518bba"
Jul 23 17:12:22.849771 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:22.849743    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:12:22.866033 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:22.866011    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jqkqj"
Jul 23 17:12:22.894444 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:22.894422    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759zx982"
Jul 23 17:12:22.918305 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:22.918276    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:12:22.951301 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:22.951280    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2pwxw\" (UniqueName: \"kubernetes.io/projected/67c1a8e8-ea9e-4d80-b4e8-cecbb603a745-kube-api-access-2pwxw\") pod \"67c1a8e8-ea9e-4d80-b4e8-cecbb603a745\" (UID: \"67c1a8e8-ea9e-4d80-b4e8-cecbb603a745\") "
Jul 23 17:12:22.951405 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:22.951337    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f0f53fe8-060d-4ddc-94a6-c361cc5d8688-bundle\") pod \"f0f53fe8-060d-4ddc-94a6-c361cc5d8688\" (UID: \"f0f53fe8-060d-4ddc-94a6-c361cc5d8688\") "
Jul 23 17:12:22.951405 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:22.951353    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/67c1a8e8-ea9e-4d80-b4e8-cecbb603a745-util\") pod \"67c1a8e8-ea9e-4d80-b4e8-cecbb603a745\" (UID: \"67c1a8e8-ea9e-4d80-b4e8-cecbb603a745\") "
Jul 23 17:12:22.951498 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:22.951407    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gh782\" (UniqueName: \"kubernetes.io/projected/f0f53fe8-060d-4ddc-94a6-c361cc5d8688-kube-api-access-gh782\") pod \"f0f53fe8-060d-4ddc-94a6-c361cc5d8688\" (UID: \"f0f53fe8-060d-4ddc-94a6-c361cc5d8688\") "
Jul 23 17:12:22.951498 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:22.951427    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f0f53fe8-060d-4ddc-94a6-c361cc5d8688-util\") pod \"f0f53fe8-060d-4ddc-94a6-c361cc5d8688\" (UID: \"f0f53fe8-060d-4ddc-94a6-c361cc5d8688\") "
Jul 23 17:12:22.951498 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:22.951475    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/67c1a8e8-ea9e-4d80-b4e8-cecbb603a745-bundle\") pod \"67c1a8e8-ea9e-4d80-b4e8-cecbb603a745\" (UID: \"67c1a8e8-ea9e-4d80-b4e8-cecbb603a745\") "
Jul 23 17:12:22.952002 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:22.951962    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0f53fe8-060d-4ddc-94a6-c361cc5d8688-bundle" (OuterVolumeSpecName: "bundle") pod "f0f53fe8-060d-4ddc-94a6-c361cc5d8688" (UID: "f0f53fe8-060d-4ddc-94a6-c361cc5d8688"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 17:12:22.952147 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:22.952008    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67c1a8e8-ea9e-4d80-b4e8-cecbb603a745-bundle" (OuterVolumeSpecName: "bundle") pod "67c1a8e8-ea9e-4d80-b4e8-cecbb603a745" (UID: "67c1a8e8-ea9e-4d80-b4e8-cecbb603a745"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 17:12:22.953885 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:22.953859    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67c1a8e8-ea9e-4d80-b4e8-cecbb603a745-kube-api-access-2pwxw" (OuterVolumeSpecName: "kube-api-access-2pwxw") pod "67c1a8e8-ea9e-4d80-b4e8-cecbb603a745" (UID: "67c1a8e8-ea9e-4d80-b4e8-cecbb603a745"). InnerVolumeSpecName "kube-api-access-2pwxw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 17:12:22.953985 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:22.953928    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0f53fe8-060d-4ddc-94a6-c361cc5d8688-kube-api-access-gh782" (OuterVolumeSpecName: "kube-api-access-gh782") pod "f0f53fe8-060d-4ddc-94a6-c361cc5d8688" (UID: "f0f53fe8-060d-4ddc-94a6-c361cc5d8688"). InnerVolumeSpecName "kube-api-access-gh782". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 17:12:22.958063 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:22.958022    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0f53fe8-060d-4ddc-94a6-c361cc5d8688-util" (OuterVolumeSpecName: "util") pod "f0f53fe8-060d-4ddc-94a6-c361cc5d8688" (UID: "f0f53fe8-060d-4ddc-94a6-c361cc5d8688"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 17:12:22.959626 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:22.959601    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67c1a8e8-ea9e-4d80-b4e8-cecbb603a745-util" (OuterVolumeSpecName: "util") pod "67c1a8e8-ea9e-4d80-b4e8-cecbb603a745" (UID: "67c1a8e8-ea9e-4d80-b4e8-cecbb603a745"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 17:12:23.053057 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:23.052973    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2pwxw\" (UniqueName: \"kubernetes.io/projected/67c1a8e8-ea9e-4d80-b4e8-cecbb603a745-kube-api-access-2pwxw\") on node \"ip-10-0-129-216.ec2.internal\" DevicePath \"\""
Jul 23 17:12:23.053057 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:23.053011    2577 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f0f53fe8-060d-4ddc-94a6-c361cc5d8688-bundle\") on node \"ip-10-0-129-216.ec2.internal\" DevicePath \"\""
Jul 23 17:12:23.053057 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:23.053023    2577 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/67c1a8e8-ea9e-4d80-b4e8-cecbb603a745-util\") on node \"ip-10-0-129-216.ec2.internal\" DevicePath \"\""
Jul 23 17:12:23.053057 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:23.053031    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gh782\" (UniqueName: \"kubernetes.io/projected/f0f53fe8-060d-4ddc-94a6-c361cc5d8688-kube-api-access-gh782\") on node \"ip-10-0-129-216.ec2.internal\" DevicePath \"\""
Jul 23 17:12:23.053057 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:23.053040    2577 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f0f53fe8-060d-4ddc-94a6-c361cc5d8688-util\") on node \"ip-10-0-129-216.ec2.internal\" DevicePath \"\""
Jul 23 17:12:23.053057 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:23.053047    2577 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/67c1a8e8-ea9e-4d80-b4e8-cecbb603a745-bundle\") on node \"ip-10-0-129-216.ec2.internal\" DevicePath \"\""
Jul 23 17:12:23.743735 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:23.743706    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759zx982"
Jul 23 17:12:23.743735 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:23.743726    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759zx982" event={"ID":"67c1a8e8-ea9e-4d80-b4e8-cecbb603a745","Type":"ContainerDied","Data":"e894fc072a4fc55c5de461f6152e583b68dbdb0ee4dd60c4514c0b7d19b8a922"}
Jul 23 17:12:23.744182 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:23.743773    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e894fc072a4fc55c5de461f6152e583b68dbdb0ee4dd60c4514c0b7d19b8a922"
Jul 23 17:12:23.745508 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:23.745486    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jqkqj" event={"ID":"f0f53fe8-060d-4ddc-94a6-c361cc5d8688","Type":"ContainerDied","Data":"99cb1180be20a5c2a4dfe220726052273f6e3835811e34dd40f47633fb281a4d"}
Jul 23 17:12:23.745508 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:23.745508    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jqkqj"
Jul 23 17:12:23.745636 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:23.745521    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="99cb1180be20a5c2a4dfe220726052273f6e3835811e34dd40f47633fb281a4d"
Jul 23 17:12:30.535061 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:30.534959    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:12:33.161465 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:33.161430    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-qtrml"]
Jul 23 17:12:33.161847 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:33.161778    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="283546e1-df7c-49a0-a72a-2318766fc257" containerName="pull"
Jul 23 17:12:33.161847 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:33.161789    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="283546e1-df7c-49a0-a72a-2318766fc257" containerName="pull"
Jul 23 17:12:33.161847 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:33.161800    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62a7ca3d-ef60-422c-bef3-627044060c5f" containerName="util"
Jul 23 17:12:33.161847 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:33.161806    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="62a7ca3d-ef60-422c-bef3-627044060c5f" containerName="util"
Jul 23 17:12:33.161847 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:33.161814    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f0f53fe8-060d-4ddc-94a6-c361cc5d8688" containerName="pull"
Jul 23 17:12:33.161847 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:33.161820    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="f0f53fe8-060d-4ddc-94a6-c361cc5d8688" containerName="pull"
Jul 23 17:12:33.161847 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:33.161826    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="67c1a8e8-ea9e-4d80-b4e8-cecbb603a745" containerName="pull"
Jul 23 17:12:33.161847 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:33.161831    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="67c1a8e8-ea9e-4d80-b4e8-cecbb603a745" containerName="pull"
Jul 23 17:12:33.161847 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:33.161843    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f0f53fe8-060d-4ddc-94a6-c361cc5d8688" containerName="util"
Jul 23 17:12:33.161847 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:33.161847    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="f0f53fe8-060d-4ddc-94a6-c361cc5d8688" containerName="util"
Jul 23 17:12:33.162189 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:33.161855    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="283546e1-df7c-49a0-a72a-2318766fc257" containerName="extract"
Jul 23 17:12:33.162189 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:33.161860    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="283546e1-df7c-49a0-a72a-2318766fc257" containerName="extract"
Jul 23 17:12:33.162189 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:33.161866    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="67c1a8e8-ea9e-4d80-b4e8-cecbb603a745" containerName="util"
Jul 23 17:12:33.162189 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:33.161871    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="67c1a8e8-ea9e-4d80-b4e8-cecbb603a745" containerName="util"
Jul 23 17:12:33.162189 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:33.161877    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62a7ca3d-ef60-422c-bef3-627044060c5f" containerName="extract"
Jul 23 17:12:33.162189 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:33.161882    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="62a7ca3d-ef60-422c-bef3-627044060c5f" containerName="extract"
Jul 23 17:12:33.162189 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:33.161887    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f0f53fe8-060d-4ddc-94a6-c361cc5d8688" containerName="extract"
Jul 23 17:12:33.162189 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:33.161893    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="f0f53fe8-060d-4ddc-94a6-c361cc5d8688" containerName="extract"
Jul 23 17:12:33.162189 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:33.161898    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62a7ca3d-ef60-422c-bef3-627044060c5f" containerName="pull"
Jul 23 17:12:33.162189 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:33.161903    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="62a7ca3d-ef60-422c-bef3-627044060c5f" containerName="pull"
Jul 23 17:12:33.162189 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:33.161915    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="283546e1-df7c-49a0-a72a-2318766fc257" containerName="util"
Jul 23 17:12:33.162189 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:33.161920    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="283546e1-df7c-49a0-a72a-2318766fc257" containerName="util"
Jul 23 17:12:33.162189 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:33.161925    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="67c1a8e8-ea9e-4d80-b4e8-cecbb603a745" containerName="extract"
Jul 23 17:12:33.162189 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:33.161930    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="67c1a8e8-ea9e-4d80-b4e8-cecbb603a745" containerName="extract"
Jul 23 17:12:33.162189 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:33.161977    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="283546e1-df7c-49a0-a72a-2318766fc257" containerName="extract"
Jul 23 17:12:33.162189 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:33.161983    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="67c1a8e8-ea9e-4d80-b4e8-cecbb603a745" containerName="extract"
Jul 23 17:12:33.162189 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:33.161989    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="62a7ca3d-ef60-422c-bef3-627044060c5f" containerName="extract"
Jul 23 17:12:33.162189 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:33.161996    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="f0f53fe8-060d-4ddc-94a6-c361cc5d8688" containerName="extract"
Jul 23 17:12:33.166932 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:33.166916    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-qtrml"
Jul 23 17:12:33.169459 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:33.169429    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Jul 23 17:12:33.169569 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:33.169465    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-qdlv6\""
Jul 23 17:12:33.177861 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:33.177838    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-qtrml"]
Jul 23 17:12:33.240871 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:33.240830    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28fkc\" (UniqueName: \"kubernetes.io/projected/35d67fdc-c2d6-4774-9912-1e088f3487bd-kube-api-access-28fkc\") pod \"dns-operator-controller-manager-864dd78c4c-qtrml\" (UID: \"35d67fdc-c2d6-4774-9912-1e088f3487bd\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-qtrml"
Jul 23 17:12:33.341271 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:33.341235    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28fkc\" (UniqueName: \"kubernetes.io/projected/35d67fdc-c2d6-4774-9912-1e088f3487bd-kube-api-access-28fkc\") pod \"dns-operator-controller-manager-864dd78c4c-qtrml\" (UID: \"35d67fdc-c2d6-4774-9912-1e088f3487bd\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-qtrml"
Jul 23 17:12:33.354680 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:33.354647    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-28fkc\" (UniqueName: \"kubernetes.io/projected/35d67fdc-c2d6-4774-9912-1e088f3487bd-kube-api-access-28fkc\") pod \"dns-operator-controller-manager-864dd78c4c-qtrml\" (UID: \"35d67fdc-c2d6-4774-9912-1e088f3487bd\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-qtrml"
Jul 23 17:12:33.478236 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:33.478201    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-qtrml"
Jul 23 17:12:33.627126 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:33.627102    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-qtrml"]
Jul 23 17:12:33.628637 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:12:33.628615    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35d67fdc_c2d6_4774_9912_1e088f3487bd.slice/crio-632af93903ae00de4bce315dce2bb02606cb427cb859a961ac97e228328c60b5 WatchSource:0}: Error finding container 632af93903ae00de4bce315dce2bb02606cb427cb859a961ac97e228328c60b5: Status 404 returned error can't find the container with id 632af93903ae00de4bce315dce2bb02606cb427cb859a961ac97e228328c60b5
Jul 23 17:12:33.800046 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:33.799972    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-qtrml" event={"ID":"35d67fdc-c2d6-4774-9912-1e088f3487bd","Type":"ContainerStarted","Data":"632af93903ae00de4bce315dce2bb02606cb427cb859a961ac97e228328c60b5"}
Jul 23 17:12:34.978312 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:34.978273    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:12:35.812429 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:35.812396    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-qtrml" event={"ID":"35d67fdc-c2d6-4774-9912-1e088f3487bd","Type":"ContainerStarted","Data":"2c9e8ffd8024a7226eee83b51ab60db9e3ea0599c8e8ddfc6266348306d6d2dd"}
Jul 23 17:12:35.812585 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:35.812453    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-qtrml"
Jul 23 17:12:35.835530 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:35.835476    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-qtrml" podStartSLOduration=0.744286537 podStartE2EDuration="2.835460472s" totalImagesPullingTime="2.091173935s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 17:12:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 17:12:35.835064505 +0000 UTC m=+451.605350042" watchObservedRunningTime="2026-07-23 17:12:35.835460472 +0000 UTC m=+451.605745991"
Jul 23 17:12:37.183879 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:37.183839    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:12:38.609211 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:38.609174    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-665gj"]
Jul 23 17:12:38.611948 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:38.611926    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-665gj"
Jul 23 17:12:38.615311 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:38.615292    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-pdv8c\""
Jul 23 17:12:38.636106 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:38.636062    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-665gj"]
Jul 23 17:12:38.688282 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:38.688240    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/fce8cece-eb0a-4e4d-aa47-79fa252c77b6-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-665gj\" (UID: \"fce8cece-eb0a-4e4d-aa47-79fa252c77b6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-665gj"
Jul 23 17:12:38.688424 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:38.688355    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9znb\" (UniqueName: \"kubernetes.io/projected/fce8cece-eb0a-4e4d-aa47-79fa252c77b6-kube-api-access-g9znb\") pod \"kuadrant-operator-controller-manager-8684c9ff49-665gj\" (UID: \"fce8cece-eb0a-4e4d-aa47-79fa252c77b6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-665gj"
Jul 23 17:12:38.780770 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:38.780735    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:12:38.788899 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:38.788871    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9znb\" (UniqueName: \"kubernetes.io/projected/fce8cece-eb0a-4e4d-aa47-79fa252c77b6-kube-api-access-g9znb\") pod \"kuadrant-operator-controller-manager-8684c9ff49-665gj\" (UID: \"fce8cece-eb0a-4e4d-aa47-79fa252c77b6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-665gj"
Jul 23 17:12:38.789039 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:38.788935    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/fce8cece-eb0a-4e4d-aa47-79fa252c77b6-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-665gj\" (UID: \"fce8cece-eb0a-4e4d-aa47-79fa252c77b6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-665gj"
Jul 23 17:12:38.789248 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:38.789228    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/fce8cece-eb0a-4e4d-aa47-79fa252c77b6-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-665gj\" (UID: \"fce8cece-eb0a-4e4d-aa47-79fa252c77b6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-665gj"
Jul 23 17:12:38.802181 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:38.802157    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9znb\" (UniqueName: \"kubernetes.io/projected/fce8cece-eb0a-4e4d-aa47-79fa252c77b6-kube-api-access-g9znb\") pod \"kuadrant-operator-controller-manager-8684c9ff49-665gj\" (UID: \"fce8cece-eb0a-4e4d-aa47-79fa252c77b6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-665gj"
Jul 23 17:12:38.921354 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:38.921282    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-665gj"
Jul 23 17:12:39.072061 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:39.072034    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-665gj"]
Jul 23 17:12:39.072503 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:12:39.072475    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfce8cece_eb0a_4e4d_aa47_79fa252c77b6.slice/crio-bb4c38b2d57ce1234b7d197af0397dfdcb3b3d64de73f657000b7630440472a9 WatchSource:0}: Error finding container bb4c38b2d57ce1234b7d197af0397dfdcb3b3d64de73f657000b7630440472a9: Status 404 returned error can't find the container with id bb4c38b2d57ce1234b7d197af0397dfdcb3b3d64de73f657000b7630440472a9
Jul 23 17:12:39.586503 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:39.586469    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:12:39.835104 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:39.835069    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-665gj" event={"ID":"fce8cece-eb0a-4e4d-aa47-79fa252c77b6","Type":"ContainerStarted","Data":"bb4c38b2d57ce1234b7d197af0397dfdcb3b3d64de73f657000b7630440472a9"}
Jul 23 17:12:41.510632 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:41.510598    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gf7xk"]
Jul 23 17:12:41.513201 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:41.513177    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gf7xk"
Jul 23 17:12:41.515611 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:41.515589    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-c9hdg\""
Jul 23 17:12:41.524464 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:41.524438    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gf7xk"]
Jul 23 17:12:41.614233 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:41.614202    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9pjv\" (UniqueName: \"kubernetes.io/projected/98a02ec9-1f64-423f-8a1a-3b0a9be58557-kube-api-access-v9pjv\") pod \"limitador-operator-controller-manager-859d7b78bd-gf7xk\" (UID: \"98a02ec9-1f64-423f-8a1a-3b0a9be58557\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gf7xk"
Jul 23 17:12:41.715153 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:41.715113    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9pjv\" (UniqueName: \"kubernetes.io/projected/98a02ec9-1f64-423f-8a1a-3b0a9be58557-kube-api-access-v9pjv\") pod \"limitador-operator-controller-manager-859d7b78bd-gf7xk\" (UID: \"98a02ec9-1f64-423f-8a1a-3b0a9be58557\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gf7xk"
Jul 23 17:12:41.728504 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:41.728464    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9pjv\" (UniqueName: \"kubernetes.io/projected/98a02ec9-1f64-423f-8a1a-3b0a9be58557-kube-api-access-v9pjv\") pod \"limitador-operator-controller-manager-859d7b78bd-gf7xk\" (UID: \"98a02ec9-1f64-423f-8a1a-3b0a9be58557\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gf7xk"
Jul 23 17:12:41.757974 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:41.757936    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:12:41.823773 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:41.823683    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gf7xk"
Jul 23 17:12:41.961328 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:41.961301    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gf7xk"]
Jul 23 17:12:41.963217 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:12:41.963191    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98a02ec9_1f64_423f_8a1a_3b0a9be58557.slice/crio-79df810df1e7ec0a599d1dc4865bc9e54dec3c50fe7a930b7db4b3c9199b9f42 WatchSource:0}: Error finding container 79df810df1e7ec0a599d1dc4865bc9e54dec3c50fe7a930b7db4b3c9199b9f42: Status 404 returned error can't find the container with id 79df810df1e7ec0a599d1dc4865bc9e54dec3c50fe7a930b7db4b3c9199b9f42
Jul 23 17:12:42.142723 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:42.142632    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:12:42.445517 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:42.445480    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:12:42.855661 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:42.855589    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gf7xk" event={"ID":"98a02ec9-1f64-423f-8a1a-3b0a9be58557","Type":"ContainerStarted","Data":"79df810df1e7ec0a599d1dc4865bc9e54dec3c50fe7a930b7db4b3c9199b9f42"}
Jul 23 17:12:44.871155 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:44.871117    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gf7xk" event={"ID":"98a02ec9-1f64-423f-8a1a-3b0a9be58557","Type":"ContainerStarted","Data":"463305a5d1b82f438b5cc817d42ae74273b65da85bae094f6dc0b6da6fc5753b"}
Jul 23 17:12:44.871814 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:44.871245    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gf7xk"
Jul 23 17:12:44.891455 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:44.891403    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gf7xk" podStartSLOduration=2.070105484 podStartE2EDuration="3.891385472s" totalImagesPullingTime="1.821279988s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 17:12:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 17:12:44.891108148 +0000 UTC m=+460.661393668" watchObservedRunningTime="2026-07-23 17:12:44.891385472 +0000 UTC m=+460.661670980"
Jul 23 17:12:45.236117 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:45.236079    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:12:45.877737 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:45.877702    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-665gj" event={"ID":"fce8cece-eb0a-4e4d-aa47-79fa252c77b6","Type":"ContainerStarted","Data":"3adc0ff7e5d0252f350579d5733a40ea4db7d1d23ff1d35ebc636d2fe642381f"}
Jul 23 17:12:45.878139 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:45.877790    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-665gj"
Jul 23 17:12:45.914612 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:45.914547    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-665gj" podStartSLOduration=1.9426020739999998 podStartE2EDuration="7.91452795s" totalImagesPullingTime="5.971925876s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 17:12:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 17:12:45.912956109 +0000 UTC m=+461.683241627" watchObservedRunningTime="2026-07-23 17:12:45.91452795 +0000 UTC m=+461.684813472"
Jul 23 17:12:46.552091 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:46.552061    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:12:46.658412 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:46.658376    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:12:46.819896 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:46.819819    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-qtrml"
Jul 23 17:12:48.104430 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:48.104394    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:12:48.205779 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:48.205748    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:12:50.156383 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:50.156354    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:12:50.285771 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:50.285740    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:12:55.880497 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:55.880466    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gf7xk"
Jul 23 17:12:56.891631 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:56.891602    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-665gj"
Jul 23 17:12:57.058437 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:57.058390    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:12:57.543340 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:57.543309    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:12:58.845646 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:58.845616    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-665gj"]
Jul 23 17:12:58.846054 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:58.845785    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-665gj" podUID="fce8cece-eb0a-4e4d-aa47-79fa252c77b6" containerName="manager" containerID="cri-o://3adc0ff7e5d0252f350579d5733a40ea4db7d1d23ff1d35ebc636d2fe642381f" gracePeriod=2
Jul 23 17:12:58.859142 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:58.859115    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-665gj"]
Jul 23 17:12:58.874110 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:12:58.874074    2577 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"fce8cece-eb0a-4e4d-aa47-79fa252c77b6\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"kuadrant-operator-controller-manager-8684c9ff49-665gj\": pods \"kuadrant-operator-controller-manager-8684c9ff49-665gj\" not found" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-665gj"
Jul 23 17:12:58.875065 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:58.875028    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2ll9d"]
Jul 23 17:12:58.875595 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:58.875571    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fce8cece-eb0a-4e4d-aa47-79fa252c77b6" containerName="manager"
Jul 23 17:12:58.875686 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:58.875598    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="fce8cece-eb0a-4e4d-aa47-79fa252c77b6" containerName="manager"
Jul 23 17:12:58.875748 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:58.875713    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="fce8cece-eb0a-4e4d-aa47-79fa252c77b6" containerName="manager"
Jul 23 17:12:58.878346 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:58.878322    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gf7xk"]
Jul 23 17:12:58.878459 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:58.878428    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2ll9d"
Jul 23 17:12:58.878525 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:58.878498    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gf7xk" podUID="98a02ec9-1f64-423f-8a1a-3b0a9be58557" containerName="manager" containerID="cri-o://463305a5d1b82f438b5cc817d42ae74273b65da85bae094f6dc0b6da6fc5753b" gracePeriod=2
Jul 23 17:12:58.881466 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:12:58.881422    2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-665gj\" is forbidden: User \"system:node:ip-10-0-129-216.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-216.ec2.internal' and this object" podUID="fce8cece-eb0a-4e4d-aa47-79fa252c77b6" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-665gj"
Jul 23 17:12:58.890841 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:58.890818    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gf7xk"]
Jul 23 17:12:58.895086 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:58.895064    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2ll9d"]
Jul 23 17:12:58.907349 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:58.907327    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-pbmw7"]
Jul 23 17:12:58.907705 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:58.907689    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="98a02ec9-1f64-423f-8a1a-3b0a9be58557" containerName="manager"
Jul 23 17:12:58.907787 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:58.907707    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="98a02ec9-1f64-423f-8a1a-3b0a9be58557" containerName="manager"
Jul 23 17:12:58.907850 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:58.907825    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="98a02ec9-1f64-423f-8a1a-3b0a9be58557" containerName="manager"
Jul 23 17:12:58.910576 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:58.910555    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-pbmw7"
Jul 23 17:12:58.919979 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:12:58.919938    2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-gf7xk\" is forbidden: User \"system:node:ip-10-0-129-216.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-216.ec2.internal' and this object" podUID="98a02ec9-1f64-423f-8a1a-3b0a9be58557" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gf7xk"
Jul 23 17:12:58.922345 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:12:58.922315    2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-gf7xk\" is forbidden: User \"system:node:ip-10-0-129-216.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-216.ec2.internal' and this object" podUID="98a02ec9-1f64-423f-8a1a-3b0a9be58557" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gf7xk"
Jul 23 17:12:58.924176 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:12:58.924147    2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-665gj\" is forbidden: User \"system:node:ip-10-0-129-216.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-216.ec2.internal' and this object" podUID="fce8cece-eb0a-4e4d-aa47-79fa252c77b6" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-665gj"
Jul 23 17:12:58.924824 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:58.924802    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-pbmw7"]
Jul 23 17:12:58.967348 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:58.967318    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dnphl\" (UniqueName: \"kubernetes.io/projected/10426c30-3cce-4885-8737-2eed66a9dab8-kube-api-access-dnphl\") pod \"kuadrant-operator-controller-manager-8684c9ff49-2ll9d\" (UID: \"10426c30-3cce-4885-8737-2eed66a9dab8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2ll9d"
Jul 23 17:12:58.967487 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:58.967351    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/10426c30-3cce-4885-8737-2eed66a9dab8-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-2ll9d\" (UID: \"10426c30-3cce-4885-8737-2eed66a9dab8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2ll9d"
Jul 23 17:12:59.068184 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:59.068152    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dnphl\" (UniqueName: \"kubernetes.io/projected/10426c30-3cce-4885-8737-2eed66a9dab8-kube-api-access-dnphl\") pod \"kuadrant-operator-controller-manager-8684c9ff49-2ll9d\" (UID: \"10426c30-3cce-4885-8737-2eed66a9dab8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2ll9d"
Jul 23 17:12:59.068338 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:59.068194    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/10426c30-3cce-4885-8737-2eed66a9dab8-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-2ll9d\" (UID: \"10426c30-3cce-4885-8737-2eed66a9dab8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2ll9d"
Jul 23 17:12:59.068338 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:59.068286    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4qrd\" (UniqueName: \"kubernetes.io/projected/f702fd02-0777-4c66-acf3-2463cd91eafe-kube-api-access-x4qrd\") pod \"limitador-operator-controller-manager-859d7b78bd-pbmw7\" (UID: \"f702fd02-0777-4c66-acf3-2463cd91eafe\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-pbmw7"
Jul 23 17:12:59.068641 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:59.068614    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/10426c30-3cce-4885-8737-2eed66a9dab8-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-2ll9d\" (UID: \"10426c30-3cce-4885-8737-2eed66a9dab8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2ll9d"
Jul 23 17:12:59.089611 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:59.089586    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dnphl\" (UniqueName: \"kubernetes.io/projected/10426c30-3cce-4885-8737-2eed66a9dab8-kube-api-access-dnphl\") pod \"kuadrant-operator-controller-manager-8684c9ff49-2ll9d\" (UID: \"10426c30-3cce-4885-8737-2eed66a9dab8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2ll9d"
Jul 23 17:12:59.119399 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:59.119378    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-665gj"
Jul 23 17:12:59.122031 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:12:59.121995    2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-665gj\" is forbidden: User \"system:node:ip-10-0-129-216.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-216.ec2.internal' and this object" podUID="fce8cece-eb0a-4e4d-aa47-79fa252c77b6" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-665gj"
Jul 23 17:12:59.122132 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:59.122101    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gf7xk"
Jul 23 17:12:59.123927 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:12:59.123908    2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-gf7xk\" is forbidden: User \"system:node:ip-10-0-129-216.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-216.ec2.internal' and this object" podUID="98a02ec9-1f64-423f-8a1a-3b0a9be58557" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gf7xk"
Jul 23 17:12:59.144071 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:12:59.144040    2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-665gj\" is forbidden: User \"system:node:ip-10-0-129-216.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-216.ec2.internal' and this object" podUID="fce8cece-eb0a-4e4d-aa47-79fa252c77b6" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-665gj"
Jul 23 17:12:59.146134 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:12:59.146115    2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-gf7xk\" is forbidden: User \"system:node:ip-10-0-129-216.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-216.ec2.internal' and this object" podUID="98a02ec9-1f64-423f-8a1a-3b0a9be58557" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gf7xk"
Jul 23 17:12:59.169230 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:59.169206    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/fce8cece-eb0a-4e4d-aa47-79fa252c77b6-extensions-socket-volume\") pod \"fce8cece-eb0a-4e4d-aa47-79fa252c77b6\" (UID: \"fce8cece-eb0a-4e4d-aa47-79fa252c77b6\") "
Jul 23 17:12:59.169358 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:59.169275    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v9pjv\" (UniqueName: \"kubernetes.io/projected/98a02ec9-1f64-423f-8a1a-3b0a9be58557-kube-api-access-v9pjv\") pod \"98a02ec9-1f64-423f-8a1a-3b0a9be58557\" (UID: \"98a02ec9-1f64-423f-8a1a-3b0a9be58557\") "
Jul 23 17:12:59.169358 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:59.169302    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g9znb\" (UniqueName: \"kubernetes.io/projected/fce8cece-eb0a-4e4d-aa47-79fa252c77b6-kube-api-access-g9znb\") pod \"fce8cece-eb0a-4e4d-aa47-79fa252c77b6\" (UID: \"fce8cece-eb0a-4e4d-aa47-79fa252c77b6\") "
Jul 23 17:12:59.169437 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:59.169412    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4qrd\" (UniqueName: \"kubernetes.io/projected/f702fd02-0777-4c66-acf3-2463cd91eafe-kube-api-access-x4qrd\") pod \"limitador-operator-controller-manager-859d7b78bd-pbmw7\" (UID: \"f702fd02-0777-4c66-acf3-2463cd91eafe\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-pbmw7"
Jul 23 17:12:59.169654 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:59.169630    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fce8cece-eb0a-4e4d-aa47-79fa252c77b6-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "fce8cece-eb0a-4e4d-aa47-79fa252c77b6" (UID: "fce8cece-eb0a-4e4d-aa47-79fa252c77b6"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 17:12:59.171393 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:59.171374    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98a02ec9-1f64-423f-8a1a-3b0a9be58557-kube-api-access-v9pjv" (OuterVolumeSpecName: "kube-api-access-v9pjv") pod "98a02ec9-1f64-423f-8a1a-3b0a9be58557" (UID: "98a02ec9-1f64-423f-8a1a-3b0a9be58557"). InnerVolumeSpecName "kube-api-access-v9pjv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 17:12:59.171460 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:59.171402    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fce8cece-eb0a-4e4d-aa47-79fa252c77b6-kube-api-access-g9znb" (OuterVolumeSpecName: "kube-api-access-g9znb") pod "fce8cece-eb0a-4e4d-aa47-79fa252c77b6" (UID: "fce8cece-eb0a-4e4d-aa47-79fa252c77b6"). InnerVolumeSpecName "kube-api-access-g9znb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 17:12:59.180913 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:59.180890    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4qrd\" (UniqueName: \"kubernetes.io/projected/f702fd02-0777-4c66-acf3-2463cd91eafe-kube-api-access-x4qrd\") pod \"limitador-operator-controller-manager-859d7b78bd-pbmw7\" (UID: \"f702fd02-0777-4c66-acf3-2463cd91eafe\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-pbmw7"
Jul 23 17:12:59.270620 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:59.270583    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g9znb\" (UniqueName: \"kubernetes.io/projected/fce8cece-eb0a-4e4d-aa47-79fa252c77b6-kube-api-access-g9znb\") on node \"ip-10-0-129-216.ec2.internal\" DevicePath \"\""
Jul 23 17:12:59.270620 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:59.270610    2577 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/fce8cece-eb0a-4e4d-aa47-79fa252c77b6-extensions-socket-volume\") on node \"ip-10-0-129-216.ec2.internal\" DevicePath \"\""
Jul 23 17:12:59.270620 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:59.270619    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v9pjv\" (UniqueName: \"kubernetes.io/projected/98a02ec9-1f64-423f-8a1a-3b0a9be58557-kube-api-access-v9pjv\") on node \"ip-10-0-129-216.ec2.internal\" DevicePath \"\""
Jul 23 17:12:59.283527 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:59.283502    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2ll9d"
Jul 23 17:12:59.289517 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:59.289500    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-pbmw7"
Jul 23 17:12:59.469971 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:59.469944    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2ll9d"]
Jul 23 17:12:59.471843 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:12:59.471813    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10426c30_3cce_4885_8737_2eed66a9dab8.slice/crio-a35438457c00cba8cbfbb47db49bf66b249480c31fd7600525a43dfcf81f21bf WatchSource:0}: Error finding container a35438457c00cba8cbfbb47db49bf66b249480c31fd7600525a43dfcf81f21bf: Status 404 returned error can't find the container with id a35438457c00cba8cbfbb47db49bf66b249480c31fd7600525a43dfcf81f21bf
Jul 23 17:12:59.491553 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:59.491533    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-pbmw7"]
Jul 23 17:12:59.492909 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:12:59.492887    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf702fd02_0777_4c66_acf3_2463cd91eafe.slice/crio-045320ff2c06532e7407eccaf23726ec58b03f225c5270dab2844a47f2f974e0 WatchSource:0}: Error finding container 045320ff2c06532e7407eccaf23726ec58b03f225c5270dab2844a47f2f974e0: Status 404 returned error can't find the container with id 045320ff2c06532e7407eccaf23726ec58b03f225c5270dab2844a47f2f974e0
Jul 23 17:12:59.960231 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:59.960196    2577 generic.go:356] "Generic (PLEG): container finished" podID="98a02ec9-1f64-423f-8a1a-3b0a9be58557" containerID="463305a5d1b82f438b5cc817d42ae74273b65da85bae094f6dc0b6da6fc5753b" exitCode=0
Jul 23 17:12:59.960677 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:59.960238    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gf7xk"
Jul 23 17:12:59.960677 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:59.960299    2577 scope.go:117] "RemoveContainer" containerID="463305a5d1b82f438b5cc817d42ae74273b65da85bae094f6dc0b6da6fc5753b"
Jul 23 17:12:59.962223 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:59.962196    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-pbmw7" event={"ID":"f702fd02-0777-4c66-acf3-2463cd91eafe","Type":"ContainerStarted","Data":"5739117105a92e530dc6d4e7edbc4d884b8c2142ccdc9ae1b9f35536ccc4c4e8"}
Jul 23 17:12:59.962367 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:59.962283    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-pbmw7" event={"ID":"f702fd02-0777-4c66-acf3-2463cd91eafe","Type":"ContainerStarted","Data":"045320ff2c06532e7407eccaf23726ec58b03f225c5270dab2844a47f2f974e0"}
Jul 23 17:12:59.962454 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:59.962382    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-pbmw7"
Jul 23 17:12:59.962576 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:12:59.962547    2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-665gj\" is forbidden: User \"system:node:ip-10-0-129-216.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-216.ec2.internal' and this object" podUID="fce8cece-eb0a-4e4d-aa47-79fa252c77b6" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-665gj"
Jul 23 17:12:59.964110 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:59.964086    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2ll9d" event={"ID":"10426c30-3cce-4885-8737-2eed66a9dab8","Type":"ContainerStarted","Data":"8fecc68665fb137ec1c40a0f8857db4691496d9e029b1d2d54aa654d45a846e7"}
Jul 23 17:12:59.964198 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:59.964132    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2ll9d" event={"ID":"10426c30-3cce-4885-8737-2eed66a9dab8","Type":"ContainerStarted","Data":"a35438457c00cba8cbfbb47db49bf66b249480c31fd7600525a43dfcf81f21bf"}
Jul 23 17:12:59.964198 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:59.964163    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2ll9d"
Jul 23 17:12:59.964420 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:12:59.964390    2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-gf7xk\" is forbidden: User \"system:node:ip-10-0-129-216.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-216.ec2.internal' and this object" podUID="98a02ec9-1f64-423f-8a1a-3b0a9be58557" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gf7xk"
Jul 23 17:12:59.965571 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:59.965542    2577 generic.go:356] "Generic (PLEG): container finished" podID="fce8cece-eb0a-4e4d-aa47-79fa252c77b6" containerID="3adc0ff7e5d0252f350579d5733a40ea4db7d1d23ff1d35ebc636d2fe642381f" exitCode=0
Jul 23 17:12:59.965649 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:59.965578    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-665gj"
Jul 23 17:12:59.966307 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:12:59.966281    2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-665gj\" is forbidden: User \"system:node:ip-10-0-129-216.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-216.ec2.internal' and this object" podUID="fce8cece-eb0a-4e4d-aa47-79fa252c77b6" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-665gj"
Jul 23 17:12:59.970915 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:59.970898    2577 scope.go:117] "RemoveContainer" containerID="463305a5d1b82f438b5cc817d42ae74273b65da85bae094f6dc0b6da6fc5753b"
Jul 23 17:12:59.971155 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:12:59.971134    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"463305a5d1b82f438b5cc817d42ae74273b65da85bae094f6dc0b6da6fc5753b\": container with ID starting with 463305a5d1b82f438b5cc817d42ae74273b65da85bae094f6dc0b6da6fc5753b not found: ID does not exist" containerID="463305a5d1b82f438b5cc817d42ae74273b65da85bae094f6dc0b6da6fc5753b"
Jul 23 17:12:59.971201 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:59.971163    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"463305a5d1b82f438b5cc817d42ae74273b65da85bae094f6dc0b6da6fc5753b"} err="failed to get container status \"463305a5d1b82f438b5cc817d42ae74273b65da85bae094f6dc0b6da6fc5753b\": rpc error: code = NotFound desc = could not find container \"463305a5d1b82f438b5cc817d42ae74273b65da85bae094f6dc0b6da6fc5753b\": container with ID starting with 463305a5d1b82f438b5cc817d42ae74273b65da85bae094f6dc0b6da6fc5753b not found: ID does not exist"
Jul 23 17:12:59.971201 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:59.971182    2577 scope.go:117] "RemoveContainer" containerID="3adc0ff7e5d0252f350579d5733a40ea4db7d1d23ff1d35ebc636d2fe642381f"
Jul 23 17:12:59.979186 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:59.979170    2577 scope.go:117] "RemoveContainer" containerID="3adc0ff7e5d0252f350579d5733a40ea4db7d1d23ff1d35ebc636d2fe642381f"
Jul 23 17:12:59.979470 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:12:59.979448    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3adc0ff7e5d0252f350579d5733a40ea4db7d1d23ff1d35ebc636d2fe642381f\": container with ID starting with 3adc0ff7e5d0252f350579d5733a40ea4db7d1d23ff1d35ebc636d2fe642381f not found: ID does not exist" containerID="3adc0ff7e5d0252f350579d5733a40ea4db7d1d23ff1d35ebc636d2fe642381f"
Jul 23 17:12:59.979540 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:59.979476    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3adc0ff7e5d0252f350579d5733a40ea4db7d1d23ff1d35ebc636d2fe642381f"} err="failed to get container status \"3adc0ff7e5d0252f350579d5733a40ea4db7d1d23ff1d35ebc636d2fe642381f\": rpc error: code = NotFound desc = could not find container \"3adc0ff7e5d0252f350579d5733a40ea4db7d1d23ff1d35ebc636d2fe642381f\": container with ID starting with 3adc0ff7e5d0252f350579d5733a40ea4db7d1d23ff1d35ebc636d2fe642381f not found: ID does not exist"
Jul 23 17:12:59.988834 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:12:59.988792    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-pbmw7" podStartSLOduration=1.988777383 podStartE2EDuration="1.988777383s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 17:12:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 17:12:59.987476335 +0000 UTC m=+475.757761854" watchObservedRunningTime="2026-07-23 17:12:59.988777383 +0000 UTC m=+475.759062915"
Jul 23 17:13:00.013716 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:13:00.013676    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2ll9d" podStartSLOduration=2.013666143 podStartE2EDuration="2.013666143s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 17:12:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 17:13:00.012541539 +0000 UTC m=+475.782827054" watchObservedRunningTime="2026-07-23 17:13:00.013666143 +0000 UTC m=+475.783951708"
Jul 23 17:13:00.014432 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:13:00.014403    2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-gf7xk\" is forbidden: User \"system:node:ip-10-0-129-216.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-216.ec2.internal' and this object" podUID="98a02ec9-1f64-423f-8a1a-3b0a9be58557" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gf7xk"
Jul 23 17:13:00.016429 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:13:00.016403    2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-gf7xk\" is forbidden: User \"system:node:ip-10-0-129-216.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-216.ec2.internal' and this object" podUID="98a02ec9-1f64-423f-8a1a-3b0a9be58557" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gf7xk"
Jul 23 17:13:00.020932 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:13:00.020906    2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-665gj\" is forbidden: User \"system:node:ip-10-0-129-216.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-216.ec2.internal' and this object" podUID="fce8cece-eb0a-4e4d-aa47-79fa252c77b6" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-665gj"
Jul 23 17:13:00.223429 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:13:00.223397    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mgqbt"]
Jul 23 17:13:00.228141 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:13:00.228126    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mgqbt"
Jul 23 17:13:00.231329 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:13:00.231290    2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-gf7xk\" is forbidden: User \"system:node:ip-10-0-129-216.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-216.ec2.internal' and this object" podUID="98a02ec9-1f64-423f-8a1a-3b0a9be58557" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gf7xk"
Jul 23 17:13:00.233373 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:13:00.233335    2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-665gj\" is forbidden: User \"system:node:ip-10-0-129-216.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-216.ec2.internal' and this object" podUID="fce8cece-eb0a-4e4d-aa47-79fa252c77b6" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-665gj"
Jul 23 17:13:00.248585 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:13:00.248553    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mgqbt"]
Jul 23 17:13:00.278861 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:13:00.278825    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9dw7\" (UniqueName: \"kubernetes.io/projected/a695e55b-cb43-455b-9eec-ddca2b0c6f99-kube-api-access-s9dw7\") pod \"kuadrant-operator-controller-manager-86878c769f-mgqbt\" (UID: \"a695e55b-cb43-455b-9eec-ddca2b0c6f99\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mgqbt"
Jul 23 17:13:00.278990 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:13:00.278902    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a695e55b-cb43-455b-9eec-ddca2b0c6f99-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-mgqbt\" (UID: \"a695e55b-cb43-455b-9eec-ddca2b0c6f99\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mgqbt"
Jul 23 17:13:00.379943 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:13:00.379896    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a695e55b-cb43-455b-9eec-ddca2b0c6f99-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-mgqbt\" (UID: \"a695e55b-cb43-455b-9eec-ddca2b0c6f99\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mgqbt"
Jul 23 17:13:00.380140 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:13:00.379987    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9dw7\" (UniqueName: \"kubernetes.io/projected/a695e55b-cb43-455b-9eec-ddca2b0c6f99-kube-api-access-s9dw7\") pod \"kuadrant-operator-controller-manager-86878c769f-mgqbt\" (UID: \"a695e55b-cb43-455b-9eec-ddca2b0c6f99\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mgqbt"
Jul 23 17:13:00.380375 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:13:00.380354    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a695e55b-cb43-455b-9eec-ddca2b0c6f99-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-mgqbt\" (UID: \"a695e55b-cb43-455b-9eec-ddca2b0c6f99\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mgqbt"
Jul 23 17:13:00.389005 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:13:00.388978    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9dw7\" (UniqueName: \"kubernetes.io/projected/a695e55b-cb43-455b-9eec-ddca2b0c6f99-kube-api-access-s9dw7\") pod \"kuadrant-operator-controller-manager-86878c769f-mgqbt\" (UID: \"a695e55b-cb43-455b-9eec-ddca2b0c6f99\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mgqbt"
Jul 23 17:13:00.539901 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:13:00.539824    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mgqbt"
Jul 23 17:13:00.678510 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:13:00.678482    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mgqbt"]
Jul 23 17:13:00.680971 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:13:00.680941    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda695e55b_cb43_455b_9eec_ddca2b0c6f99.slice/crio-3c8d57a040db065b1d87ba96c2db577e9aa37db2eef9d0e424a2f3b926c3525a WatchSource:0}: Error finding container 3c8d57a040db065b1d87ba96c2db577e9aa37db2eef9d0e424a2f3b926c3525a: Status 404 returned error can't find the container with id 3c8d57a040db065b1d87ba96c2db577e9aa37db2eef9d0e424a2f3b926c3525a
Jul 23 17:13:00.765893 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:13:00.765861    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="98a02ec9-1f64-423f-8a1a-3b0a9be58557" path="/var/lib/kubelet/pods/98a02ec9-1f64-423f-8a1a-3b0a9be58557/volumes"
Jul 23 17:13:00.766197 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:13:00.766184    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fce8cece-eb0a-4e4d-aa47-79fa252c77b6" path="/var/lib/kubelet/pods/fce8cece-eb0a-4e4d-aa47-79fa252c77b6/volumes"
Jul 23 17:13:00.972781 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:13:00.972735    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mgqbt" event={"ID":"a695e55b-cb43-455b-9eec-ddca2b0c6f99","Type":"ContainerStarted","Data":"8ef36b1cad084b860a163955bd3806b49cd59e3c4166d5d85a887bd26f672ffa"}
Jul 23 17:13:00.973213 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:13:00.972829    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mgqbt" event={"ID":"a695e55b-cb43-455b-9eec-ddca2b0c6f99","Type":"ContainerStarted","Data":"3c8d57a040db065b1d87ba96c2db577e9aa37db2eef9d0e424a2f3b926c3525a"}
Jul 23 17:13:00.973213 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:13:00.972949    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mgqbt"
Jul 23 17:13:01.000665 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:13:01.000623    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mgqbt" podStartSLOduration=1.000607506 podStartE2EDuration="1.000607506s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 17:13:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 17:13:00.999347811 +0000 UTC m=+476.769633329" watchObservedRunningTime="2026-07-23 17:13:01.000607506 +0000 UTC m=+476.770893037"
Jul 23 17:13:01.829820 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:13:01.829790    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:13:01.926660 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:13:01.926629    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:13:04.405678 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:13:04.405633    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:13:04.505774 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:13:04.505735    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:13:07.804376 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:13:07.804339    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:13:07.901825 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:13:07.901791    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:13:10.976397 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:13:10.976364    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2ll9d"
Jul 23 17:13:10.976758 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:13:10.976435    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-pbmw7"
Jul 23 17:13:11.982120 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:13:11.982083    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mgqbt"
Jul 23 17:13:12.068380 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:13:12.068347    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2ll9d"]
Jul 23 17:13:12.068580 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:13:12.068557    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2ll9d" podUID="10426c30-3cce-4885-8737-2eed66a9dab8" containerName="manager" containerID="cri-o://8fecc68665fb137ec1c40a0f8857db4691496d9e029b1d2d54aa654d45a846e7" gracePeriod=10
Jul 23 17:13:12.318041 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:13:12.318017    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2ll9d"
Jul 23 17:13:12.383887 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:13:12.383860    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dnphl\" (UniqueName: \"kubernetes.io/projected/10426c30-3cce-4885-8737-2eed66a9dab8-kube-api-access-dnphl\") pod \"10426c30-3cce-4885-8737-2eed66a9dab8\" (UID: \"10426c30-3cce-4885-8737-2eed66a9dab8\") "
Jul 23 17:13:12.384044 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:13:12.383929    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/10426c30-3cce-4885-8737-2eed66a9dab8-extensions-socket-volume\") pod \"10426c30-3cce-4885-8737-2eed66a9dab8\" (UID: \"10426c30-3cce-4885-8737-2eed66a9dab8\") "
Jul 23 17:13:12.384338 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:13:12.384314    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10426c30-3cce-4885-8737-2eed66a9dab8-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "10426c30-3cce-4885-8737-2eed66a9dab8" (UID: "10426c30-3cce-4885-8737-2eed66a9dab8"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 17:13:12.385951 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:13:12.385925    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10426c30-3cce-4885-8737-2eed66a9dab8-kube-api-access-dnphl" (OuterVolumeSpecName: "kube-api-access-dnphl") pod "10426c30-3cce-4885-8737-2eed66a9dab8" (UID: "10426c30-3cce-4885-8737-2eed66a9dab8"). InnerVolumeSpecName "kube-api-access-dnphl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 17:13:12.485625 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:13:12.485538    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dnphl\" (UniqueName: \"kubernetes.io/projected/10426c30-3cce-4885-8737-2eed66a9dab8-kube-api-access-dnphl\") on node \"ip-10-0-129-216.ec2.internal\" DevicePath \"\""
Jul 23 17:13:12.485625 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:13:12.485572    2577 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/10426c30-3cce-4885-8737-2eed66a9dab8-extensions-socket-volume\") on node \"ip-10-0-129-216.ec2.internal\" DevicePath \"\""
Jul 23 17:13:13.039204 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:13:13.039115    2577 generic.go:356] "Generic (PLEG): container finished" podID="10426c30-3cce-4885-8737-2eed66a9dab8" containerID="8fecc68665fb137ec1c40a0f8857db4691496d9e029b1d2d54aa654d45a846e7" exitCode=0
Jul 23 17:13:13.039204 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:13:13.039164    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2ll9d"
Jul 23 17:13:13.039733 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:13:13.039200    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2ll9d" event={"ID":"10426c30-3cce-4885-8737-2eed66a9dab8","Type":"ContainerDied","Data":"8fecc68665fb137ec1c40a0f8857db4691496d9e029b1d2d54aa654d45a846e7"}
Jul 23 17:13:13.039733 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:13:13.039268    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2ll9d" event={"ID":"10426c30-3cce-4885-8737-2eed66a9dab8","Type":"ContainerDied","Data":"a35438457c00cba8cbfbb47db49bf66b249480c31fd7600525a43dfcf81f21bf"}
Jul 23 17:13:13.039733 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:13:13.039299    2577 scope.go:117] "RemoveContainer" containerID="8fecc68665fb137ec1c40a0f8857db4691496d9e029b1d2d54aa654d45a846e7"
Jul 23 17:13:13.048085 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:13:13.048069    2577 scope.go:117] "RemoveContainer" containerID="8fecc68665fb137ec1c40a0f8857db4691496d9e029b1d2d54aa654d45a846e7"
Jul 23 17:13:13.048416 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:13:13.048395    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8fecc68665fb137ec1c40a0f8857db4691496d9e029b1d2d54aa654d45a846e7\": container with ID starting with 8fecc68665fb137ec1c40a0f8857db4691496d9e029b1d2d54aa654d45a846e7 not found: ID does not exist" containerID="8fecc68665fb137ec1c40a0f8857db4691496d9e029b1d2d54aa654d45a846e7"
Jul 23 17:13:13.048518 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:13:13.048422    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8fecc68665fb137ec1c40a0f8857db4691496d9e029b1d2d54aa654d45a846e7"} err="failed to get container status \"8fecc68665fb137ec1c40a0f8857db4691496d9e029b1d2d54aa654d45a846e7\": rpc error: code = NotFound desc = could not find container \"8fecc68665fb137ec1c40a0f8857db4691496d9e029b1d2d54aa654d45a846e7\": container with ID starting with 8fecc68665fb137ec1c40a0f8857db4691496d9e029b1d2d54aa654d45a846e7 not found: ID does not exist"
Jul 23 17:13:13.063156 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:13:13.063134    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2ll9d"]
Jul 23 17:13:13.067785 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:13:13.067763    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2ll9d"]
Jul 23 17:13:14.769297 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:13:14.769193    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="10426c30-3cce-4885-8737-2eed66a9dab8" path="/var/lib/kubelet/pods/10426c30-3cce-4885-8737-2eed66a9dab8/volumes"
Jul 23 17:13:14.816186 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:13:14.816151    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:13:14.919843 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:13:14.919816    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:13:16.024865 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:13:16.024830    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:13:16.125391 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:13:16.125360    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:13:19.614642 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:13:19.614609    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:13:19.715755 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:13:19.715723    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:13:22.226411 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:13:22.226374    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:13:22.325975 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:13:22.325942    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:13:25.822590 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:13:25.822555    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:13:25.922497 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:13:25.922466    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:13:26.023219 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:13:26.023189    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:13:26.178764 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:13:26.178680    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:13:26.283302 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:13:26.283264    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:13:32.757893 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:13:32.757858    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:13:33.099378 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:13:33.099298    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:13:33.453040 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:13:33.453000    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:13:33.556240 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:13:33.556209    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:13:34.926430 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:13:34.926391    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:13:34.941139 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:13:34.941105    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-t7n57"]
Jul 23 17:13:34.941522 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:13:34.941498    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="10426c30-3cce-4885-8737-2eed66a9dab8" containerName="manager"
Jul 23 17:13:34.941522 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:13:34.941513    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="10426c30-3cce-4885-8737-2eed66a9dab8" containerName="manager"
Jul 23 17:13:34.941683 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:13:34.941630    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="10426c30-3cce-4885-8737-2eed66a9dab8" containerName="manager"
Jul 23 17:13:34.943790 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:13:34.943771    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-t7n57"
Jul 23 17:13:34.945953 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:13:34.945935    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Jul 23 17:13:34.946064 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:13:34.945934    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-qbxzr\""
Jul 23 17:13:34.955158 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:13:34.955138    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-t7n57"]
Jul 23 17:13:35.037446 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:13:35.037419    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:13:35.038403 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:13:35.038378    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-t7n57"]
Jul 23 17:13:35.084333 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:13:35.084308    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/f8705db3-c732-47d1-884f-7001730408aa-config-file\") pod \"limitador-limitador-64c94fd64f-t7n57\" (UID: \"f8705db3-c732-47d1-884f-7001730408aa\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-t7n57"
Jul 23 17:13:35.084461 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:13:35.084437    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lnggp\" (UniqueName: \"kubernetes.io/projected/f8705db3-c732-47d1-884f-7001730408aa-kube-api-access-lnggp\") pod \"limitador-limitador-64c94fd64f-t7n57\" (UID: \"f8705db3-c732-47d1-884f-7001730408aa\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-t7n57"
Jul 23 17:13:35.134338 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:13:35.134305    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:13:35.184904 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:13:35.184822    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lnggp\" (UniqueName: \"kubernetes.io/projected/f8705db3-c732-47d1-884f-7001730408aa-kube-api-access-lnggp\") pod \"limitador-limitador-64c94fd64f-t7n57\" (UID: \"f8705db3-c732-47d1-884f-7001730408aa\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-t7n57"
Jul 23 17:13:35.184904 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:13:35.184861    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/f8705db3-c732-47d1-884f-7001730408aa-config-file\") pod \"limitador-limitador-64c94fd64f-t7n57\" (UID: \"f8705db3-c732-47d1-884f-7001730408aa\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-t7n57"
Jul 23 17:13:35.185517 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:13:35.185492    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/f8705db3-c732-47d1-884f-7001730408aa-config-file\") pod \"limitador-limitador-64c94fd64f-t7n57\" (UID: \"f8705db3-c732-47d1-884f-7001730408aa\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-t7n57"
Jul 23 17:13:35.194758 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:13:35.194733    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lnggp\" (UniqueName: \"kubernetes.io/projected/f8705db3-c732-47d1-884f-7001730408aa-kube-api-access-lnggp\") pod \"limitador-limitador-64c94fd64f-t7n57\" (UID: \"f8705db3-c732-47d1-884f-7001730408aa\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-t7n57"
Jul 23 17:13:35.254533 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:13:35.254505    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-t7n57"
Jul 23 17:13:35.392158 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:13:35.392135    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-t7n57"]
Jul 23 17:13:35.393836 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:13:35.393808    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8705db3_c732_47d1_884f_7001730408aa.slice/crio-e37d4fc376925f700828cd9c6bf04fc737ea71c1034868c1023234cf70534bf8 WatchSource:0}: Error finding container e37d4fc376925f700828cd9c6bf04fc737ea71c1034868c1023234cf70534bf8: Status 404 returned error can't find the container with id e37d4fc376925f700828cd9c6bf04fc737ea71c1034868c1023234cf70534bf8
Jul 23 17:13:35.776767 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:13:35.776730    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-58cd955675-p7jvg"]
Jul 23 17:13:35.780162 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:13:35.780142    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-p7jvg"
Jul 23 17:13:35.784166 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:13:35.784142    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-7gscs\""
Jul 23 17:13:35.793959 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:13:35.793936    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-p7jvg"]
Jul 23 17:13:35.812181 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:13:35.812150    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:13:35.885695 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:13:35.885667    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:13:35.890888 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:13:35.890865    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hm7xp\" (UniqueName: \"kubernetes.io/projected/ae36ad4c-84e7-409a-8c5c-0b500b3f0781-kube-api-access-hm7xp\") pod \"authorino-58cd955675-p7jvg\" (UID: \"ae36ad4c-84e7-409a-8c5c-0b500b3f0781\") " pod="kuadrant-system/authorino-58cd955675-p7jvg"
Jul 23 17:13:35.991518 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:13:35.991488    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hm7xp\" (UniqueName: \"kubernetes.io/projected/ae36ad4c-84e7-409a-8c5c-0b500b3f0781-kube-api-access-hm7xp\") pod \"authorino-58cd955675-p7jvg\" (UID: \"ae36ad4c-84e7-409a-8c5c-0b500b3f0781\") " pod="kuadrant-system/authorino-58cd955675-p7jvg"
Jul 23 17:13:36.000165 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:13:36.000137    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hm7xp\" (UniqueName: \"kubernetes.io/projected/ae36ad4c-84e7-409a-8c5c-0b500b3f0781-kube-api-access-hm7xp\") pod \"authorino-58cd955675-p7jvg\" (UID: \"ae36ad4c-84e7-409a-8c5c-0b500b3f0781\") " pod="kuadrant-system/authorino-58cd955675-p7jvg"
Jul 23 17:13:36.089836 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:13:36.089770    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-p7jvg"
Jul 23 17:13:36.164611 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:13:36.164574    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-t7n57" event={"ID":"f8705db3-c732-47d1-884f-7001730408aa","Type":"ContainerStarted","Data":"e37d4fc376925f700828cd9c6bf04fc737ea71c1034868c1023234cf70534bf8"}
Jul 23 17:13:36.262655 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:13:36.262628    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-p7jvg"]
Jul 23 17:13:37.175015 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:13:37.174985    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-p7jvg" event={"ID":"ae36ad4c-84e7-409a-8c5c-0b500b3f0781","Type":"ContainerStarted","Data":"ea7f446f22054c9cbd1dabdb29689f8c09898b6789fb90d6451788678fae7eae"}
Jul 23 17:13:38.050438 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:13:38.050391    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:13:38.146957 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:13:38.146921    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:13:40.195824 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:13:40.195793    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-p7jvg" event={"ID":"ae36ad4c-84e7-409a-8c5c-0b500b3f0781","Type":"ContainerStarted","Data":"4c76692a5001ea6615e3c3a8f44b45a41a3237be7f0e5d123016cd6c7c9dfdaf"}
Jul 23 17:13:40.197738 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:13:40.197694    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-t7n57" event={"ID":"f8705db3-c732-47d1-884f-7001730408aa","Type":"ContainerStarted","Data":"14493f317b7aac40ceab870ed81126d4690531ad9a40629d7bf7988a5b246b2a"}
Jul 23 17:13:40.197925 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:13:40.197846    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-t7n57"
Jul 23 17:13:40.212057 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:13:40.212007    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-58cd955675-p7jvg" podStartSLOduration=1.435796248 podStartE2EDuration="5.21199145s" totalImagesPullingTime="3.776195202s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 17:13:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 17:13:40.210096314 +0000 UTC m=+515.980381834" watchObservedRunningTime="2026-07-23 17:13:40.21199145 +0000 UTC m=+515.982276968"
Jul 23 17:13:40.227380 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:13:40.227327    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-64c94fd64f-t7n57" podStartSLOduration=1.534003651 podStartE2EDuration="6.227307787s" totalImagesPullingTime="4.693304136s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 17:13:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 17:13:40.226699332 +0000 UTC m=+515.996984852" watchObservedRunningTime="2026-07-23 17:13:40.227307787 +0000 UTC m=+515.997593309"
Jul 23 17:13:41.928393 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:13:41.928352    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-p7jvg"]
Jul 23 17:13:42.208777 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:13:42.208738    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-58cd955675-p7jvg" podUID="ae36ad4c-84e7-409a-8c5c-0b500b3f0781" containerName="authorino" containerID="cri-o://4c76692a5001ea6615e3c3a8f44b45a41a3237be7f0e5d123016cd6c7c9dfdaf" gracePeriod=30
Jul 23 17:13:42.459803 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:13:42.459748    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-p7jvg"
Jul 23 17:13:42.552137 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:13:42.552110    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hm7xp\" (UniqueName: \"kubernetes.io/projected/ae36ad4c-84e7-409a-8c5c-0b500b3f0781-kube-api-access-hm7xp\") pod \"ae36ad4c-84e7-409a-8c5c-0b500b3f0781\" (UID: \"ae36ad4c-84e7-409a-8c5c-0b500b3f0781\") "
Jul 23 17:13:42.554136 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:13:42.554110    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae36ad4c-84e7-409a-8c5c-0b500b3f0781-kube-api-access-hm7xp" (OuterVolumeSpecName: "kube-api-access-hm7xp") pod "ae36ad4c-84e7-409a-8c5c-0b500b3f0781" (UID: "ae36ad4c-84e7-409a-8c5c-0b500b3f0781"). InnerVolumeSpecName "kube-api-access-hm7xp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 17:13:42.653685 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:13:42.653657    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hm7xp\" (UniqueName: \"kubernetes.io/projected/ae36ad4c-84e7-409a-8c5c-0b500b3f0781-kube-api-access-hm7xp\") on node \"ip-10-0-129-216.ec2.internal\" DevicePath \"\""
Jul 23 17:13:43.214924 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:13:43.214889    2577 generic.go:356] "Generic (PLEG): container finished" podID="ae36ad4c-84e7-409a-8c5c-0b500b3f0781" containerID="4c76692a5001ea6615e3c3a8f44b45a41a3237be7f0e5d123016cd6c7c9dfdaf" exitCode=0
Jul 23 17:13:43.215355 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:13:43.214936    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-p7jvg"
Jul 23 17:13:43.215355 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:13:43.214978    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-p7jvg" event={"ID":"ae36ad4c-84e7-409a-8c5c-0b500b3f0781","Type":"ContainerDied","Data":"4c76692a5001ea6615e3c3a8f44b45a41a3237be7f0e5d123016cd6c7c9dfdaf"}
Jul 23 17:13:43.215355 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:13:43.215027    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-p7jvg" event={"ID":"ae36ad4c-84e7-409a-8c5c-0b500b3f0781","Type":"ContainerDied","Data":"ea7f446f22054c9cbd1dabdb29689f8c09898b6789fb90d6451788678fae7eae"}
Jul 23 17:13:43.215355 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:13:43.215045    2577 scope.go:117] "RemoveContainer" containerID="4c76692a5001ea6615e3c3a8f44b45a41a3237be7f0e5d123016cd6c7c9dfdaf"
Jul 23 17:13:43.223711 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:13:43.223691    2577 scope.go:117] "RemoveContainer" containerID="4c76692a5001ea6615e3c3a8f44b45a41a3237be7f0e5d123016cd6c7c9dfdaf"
Jul 23 17:13:43.223993 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:13:43.223950    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c76692a5001ea6615e3c3a8f44b45a41a3237be7f0e5d123016cd6c7c9dfdaf\": container with ID starting with 4c76692a5001ea6615e3c3a8f44b45a41a3237be7f0e5d123016cd6c7c9dfdaf not found: ID does not exist" containerID="4c76692a5001ea6615e3c3a8f44b45a41a3237be7f0e5d123016cd6c7c9dfdaf"
Jul 23 17:13:43.224035 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:13:43.224002    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c76692a5001ea6615e3c3a8f44b45a41a3237be7f0e5d123016cd6c7c9dfdaf"} err="failed to get container status \"4c76692a5001ea6615e3c3a8f44b45a41a3237be7f0e5d123016cd6c7c9dfdaf\": rpc error: code = NotFound desc = could not find container \"4c76692a5001ea6615e3c3a8f44b45a41a3237be7f0e5d123016cd6c7c9dfdaf\": container with ID starting with 4c76692a5001ea6615e3c3a8f44b45a41a3237be7f0e5d123016cd6c7c9dfdaf not found: ID does not exist"
Jul 23 17:13:43.243204 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:13:43.243179    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-p7jvg"]
Jul 23 17:13:43.247425 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:13:43.247401    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-58cd955675-p7jvg"]
Jul 23 17:13:44.765629 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:13:44.765589    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ae36ad4c-84e7-409a-8c5c-0b500b3f0781" path="/var/lib/kubelet/pods/ae36ad4c-84e7-409a-8c5c-0b500b3f0781/volumes"
Jul 23 17:13:51.203737 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:13:51.203705    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-t7n57"
Jul 23 17:13:52.943284 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:13:52.943230    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-t7n57"]
Jul 23 17:13:52.943685 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:13:52.943456    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-64c94fd64f-t7n57" podUID="f8705db3-c732-47d1-884f-7001730408aa" containerName="limitador" containerID="cri-o://14493f317b7aac40ceab870ed81126d4690531ad9a40629d7bf7988a5b246b2a" gracePeriod=30
Jul 23 17:13:53.493178 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:13:53.493148    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-t7n57"
Jul 23 17:13:53.540951 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:13:53.540878    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lnggp\" (UniqueName: \"kubernetes.io/projected/f8705db3-c732-47d1-884f-7001730408aa-kube-api-access-lnggp\") pod \"f8705db3-c732-47d1-884f-7001730408aa\" (UID: \"f8705db3-c732-47d1-884f-7001730408aa\") "
Jul 23 17:13:53.540951 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:13:53.540919    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/f8705db3-c732-47d1-884f-7001730408aa-config-file\") pod \"f8705db3-c732-47d1-884f-7001730408aa\" (UID: \"f8705db3-c732-47d1-884f-7001730408aa\") "
Jul 23 17:13:53.541334 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:13:53.541307    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8705db3-c732-47d1-884f-7001730408aa-config-file" (OuterVolumeSpecName: "config-file") pod "f8705db3-c732-47d1-884f-7001730408aa" (UID: "f8705db3-c732-47d1-884f-7001730408aa"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 17:13:53.542920 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:13:53.542893    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8705db3-c732-47d1-884f-7001730408aa-kube-api-access-lnggp" (OuterVolumeSpecName: "kube-api-access-lnggp") pod "f8705db3-c732-47d1-884f-7001730408aa" (UID: "f8705db3-c732-47d1-884f-7001730408aa"). InnerVolumeSpecName "kube-api-access-lnggp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 17:13:53.641677 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:13:53.641638    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lnggp\" (UniqueName: \"kubernetes.io/projected/f8705db3-c732-47d1-884f-7001730408aa-kube-api-access-lnggp\") on node \"ip-10-0-129-216.ec2.internal\" DevicePath \"\""
Jul 23 17:13:53.641677 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:13:53.641669    2577 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/f8705db3-c732-47d1-884f-7001730408aa-config-file\") on node \"ip-10-0-129-216.ec2.internal\" DevicePath \"\""
Jul 23 17:13:54.273219 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:13:54.273181    2577 generic.go:356] "Generic (PLEG): container finished" podID="f8705db3-c732-47d1-884f-7001730408aa" containerID="14493f317b7aac40ceab870ed81126d4690531ad9a40629d7bf7988a5b246b2a" exitCode=0
Jul 23 17:13:54.273691 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:13:54.273248    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-t7n57"
Jul 23 17:13:54.273691 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:13:54.273291    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-t7n57" event={"ID":"f8705db3-c732-47d1-884f-7001730408aa","Type":"ContainerDied","Data":"14493f317b7aac40ceab870ed81126d4690531ad9a40629d7bf7988a5b246b2a"}
Jul 23 17:13:54.273691 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:13:54.273352    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-t7n57" event={"ID":"f8705db3-c732-47d1-884f-7001730408aa","Type":"ContainerDied","Data":"e37d4fc376925f700828cd9c6bf04fc737ea71c1034868c1023234cf70534bf8"}
Jul 23 17:13:54.273691 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:13:54.273375    2577 scope.go:117] "RemoveContainer" containerID="14493f317b7aac40ceab870ed81126d4690531ad9a40629d7bf7988a5b246b2a"
Jul 23 17:13:54.282225 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:13:54.282199    2577 scope.go:117] "RemoveContainer" containerID="14493f317b7aac40ceab870ed81126d4690531ad9a40629d7bf7988a5b246b2a"
Jul 23 17:13:54.282508 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:13:54.282489    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14493f317b7aac40ceab870ed81126d4690531ad9a40629d7bf7988a5b246b2a\": container with ID starting with 14493f317b7aac40ceab870ed81126d4690531ad9a40629d7bf7988a5b246b2a not found: ID does not exist" containerID="14493f317b7aac40ceab870ed81126d4690531ad9a40629d7bf7988a5b246b2a"
Jul 23 17:13:54.282561 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:13:54.282515    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14493f317b7aac40ceab870ed81126d4690531ad9a40629d7bf7988a5b246b2a"} err="failed to get container status \"14493f317b7aac40ceab870ed81126d4690531ad9a40629d7bf7988a5b246b2a\": rpc error: code = NotFound desc = could not find container \"14493f317b7aac40ceab870ed81126d4690531ad9a40629d7bf7988a5b246b2a\": container with ID starting with 14493f317b7aac40ceab870ed81126d4690531ad9a40629d7bf7988a5b246b2a not found: ID does not exist"
Jul 23 17:13:54.294591 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:13:54.294567    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-t7n57"]
Jul 23 17:13:54.298015 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:13:54.297994    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-t7n57"]
Jul 23 17:13:54.766792 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:13:54.766752    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f8705db3-c732-47d1-884f-7001730408aa" path="/var/lib/kubelet/pods/f8705db3-c732-47d1-884f-7001730408aa/volumes"
Jul 23 17:13:56.238741 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:13:56.238707    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:13:56.325714 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:13:56.325681    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:13:56.682609 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:13:56.682507    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-llvh2"]
Jul 23 17:13:56.682860 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:13:56.682848    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae36ad4c-84e7-409a-8c5c-0b500b3f0781" containerName="authorino"
Jul 23 17:13:56.682915 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:13:56.682862    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae36ad4c-84e7-409a-8c5c-0b500b3f0781" containerName="authorino"
Jul 23 17:13:56.682915 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:13:56.682887    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8705db3-c732-47d1-884f-7001730408aa" containerName="limitador"
Jul 23 17:13:56.682915 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:13:56.682892    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8705db3-c732-47d1-884f-7001730408aa" containerName="limitador"
Jul 23 17:13:56.683014 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:13:56.682945    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="ae36ad4c-84e7-409a-8c5c-0b500b3f0781" containerName="authorino"
Jul 23 17:13:56.683014 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:13:56.682954    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="f8705db3-c732-47d1-884f-7001730408aa" containerName="limitador"
Jul 23 17:13:56.687104 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:13:56.687085    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-llvh2"
Jul 23 17:13:56.690130 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:13:56.690107    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 23 17:13:56.690494 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:13:56.690248    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 23 17:13:56.690978 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:13:56.690962    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\""
Jul 23 17:13:56.691064 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:13:56.690967    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-nc7gg\""
Jul 23 17:13:56.696697 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:13:56.696672    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-llvh2"]
Jul 23 17:13:56.728304 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:13:56.728273    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:13:56.764312 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:13:56.764286    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/983722e8-323d-4894-9391-513f0692387e-data\") pod \"postgres-69fb4594cf-llvh2\" (UID: \"983722e8-323d-4894-9391-513f0692387e\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-llvh2"
Jul 23 17:13:56.764412 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:13:56.764322    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-578pc\" (UniqueName: \"kubernetes.io/projected/983722e8-323d-4894-9391-513f0692387e-kube-api-access-578pc\") pod \"postgres-69fb4594cf-llvh2\" (UID: \"983722e8-323d-4894-9391-513f0692387e\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-llvh2"
Jul 23 17:13:56.865428 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:13:56.865390    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/983722e8-323d-4894-9391-513f0692387e-data\") pod \"postgres-69fb4594cf-llvh2\" (UID: \"983722e8-323d-4894-9391-513f0692387e\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-llvh2"
Jul 23 17:13:56.865596 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:13:56.865443    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-578pc\" (UniqueName: \"kubernetes.io/projected/983722e8-323d-4894-9391-513f0692387e-kube-api-access-578pc\") pod \"postgres-69fb4594cf-llvh2\" (UID: \"983722e8-323d-4894-9391-513f0692387e\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-llvh2"
Jul 23 17:13:56.865846 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:13:56.865826    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/983722e8-323d-4894-9391-513f0692387e-data\") pod \"postgres-69fb4594cf-llvh2\" (UID: \"983722e8-323d-4894-9391-513f0692387e\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-llvh2"
Jul 23 17:13:56.878827 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:13:56.878802    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-578pc\" (UniqueName: \"kubernetes.io/projected/983722e8-323d-4894-9391-513f0692387e-kube-api-access-578pc\") pod \"postgres-69fb4594cf-llvh2\" (UID: \"983722e8-323d-4894-9391-513f0692387e\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-llvh2"
Jul 23 17:13:56.998461 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:13:56.998431    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-llvh2"
Jul 23 17:13:57.130816 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:13:57.130791    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-llvh2"]
Jul 23 17:13:57.132495 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:13:57.132461    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod983722e8_323d_4894_9391_513f0692387e.slice/crio-753f97b4fa91d19418dc77a5f3bbca41201883d9b35073ad6d6c78b801d41bff WatchSource:0}: Error finding container 753f97b4fa91d19418dc77a5f3bbca41201883d9b35073ad6d6c78b801d41bff: Status 404 returned error can't find the container with id 753f97b4fa91d19418dc77a5f3bbca41201883d9b35073ad6d6c78b801d41bff
Jul 23 17:13:57.291846 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:13:57.291769    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-llvh2" event={"ID":"983722e8-323d-4894-9391-513f0692387e","Type":"ContainerStarted","Data":"753f97b4fa91d19418dc77a5f3bbca41201883d9b35073ad6d6c78b801d41bff"}
Jul 23 17:14:02.325974 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:14:02.325945    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-llvh2" event={"ID":"983722e8-323d-4894-9391-513f0692387e","Type":"ContainerStarted","Data":"d170309453f9787807bc749a941e1afdf7b1cb895fe3b0bbd2792d58c6b53d5f"}
Jul 23 17:14:02.326372 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:14:02.326016    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-llvh2"
Jul 23 17:14:02.342483 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:14:02.342441    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-llvh2" podStartSLOduration=1.2504725780000001 podStartE2EDuration="6.342425428s" totalImagesPullingTime="5.09195285s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 17:13:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 17:14:02.34146201 +0000 UTC m=+538.111747529" watchObservedRunningTime="2026-07-23 17:14:02.342425428 +0000 UTC m=+538.112710993"
Jul 23 17:14:05.165347 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:14:05.165315    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:14:08.360615 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:14:08.360588    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-llvh2"
Jul 23 17:14:08.691391 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:14:08.691356    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:14:08.991316 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:14:08.991234    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:14:09.013875 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:14:09.013842    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5d4b67c446-7q22k"]
Jul 23 17:14:09.019633 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:14:09.019598    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-7q22k"
Jul 23 17:14:09.023437 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:14:09.023412    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-7gscs\""
Jul 23 17:14:09.075479 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:14:09.075447    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-77nc7\" (UniqueName: \"kubernetes.io/projected/42d626ea-e4bb-439a-83cf-b60dd2f5dff4-kube-api-access-77nc7\") pod \"authorino-5d4b67c446-7q22k\" (UID: \"42d626ea-e4bb-439a-83cf-b60dd2f5dff4\") " pod="kuadrant-system/authorino-5d4b67c446-7q22k"
Jul 23 17:14:09.081632 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:14:09.081603    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-7q22k"]
Jul 23 17:14:09.175989 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:14:09.175958    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77nc7\" (UniqueName: \"kubernetes.io/projected/42d626ea-e4bb-439a-83cf-b60dd2f5dff4-kube-api-access-77nc7\") pod \"authorino-5d4b67c446-7q22k\" (UID: \"42d626ea-e4bb-439a-83cf-b60dd2f5dff4\") " pod="kuadrant-system/authorino-5d4b67c446-7q22k"
Jul 23 17:14:09.184708 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:14:09.184674    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-77nc7\" (UniqueName: \"kubernetes.io/projected/42d626ea-e4bb-439a-83cf-b60dd2f5dff4-kube-api-access-77nc7\") pod \"authorino-5d4b67c446-7q22k\" (UID: \"42d626ea-e4bb-439a-83cf-b60dd2f5dff4\") " pod="kuadrant-system/authorino-5d4b67c446-7q22k"
Jul 23 17:14:09.233663 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:14:09.233628    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-7q22k"]
Jul 23 17:14:09.233907 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:14:09.233891    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-7q22k"
Jul 23 17:14:09.264052 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:14:09.264025    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6c79785f8f-ggwxc"]
Jul 23 17:14:09.268419 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:14:09.268402    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6c79785f8f-ggwxc"
Jul 23 17:14:09.277238 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:14:09.277214    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6c79785f8f-ggwxc"]
Jul 23 17:14:09.367131 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:14:09.367109    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-7q22k"]
Jul 23 17:14:09.368849 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:14:09.368813    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod42d626ea_e4bb_439a_83cf_b60dd2f5dff4.slice/crio-1e7ec4cb1ddf61ae1a5205b9a1182d5370c406b60cd512b18255a68db21f8217 WatchSource:0}: Error finding container 1e7ec4cb1ddf61ae1a5205b9a1182d5370c406b60cd512b18255a68db21f8217: Status 404 returned error can't find the container with id 1e7ec4cb1ddf61ae1a5205b9a1182d5370c406b60cd512b18255a68db21f8217
Jul 23 17:14:09.377824 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:14:09.377792    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fkpgq\" (UniqueName: \"kubernetes.io/projected/0f83ac89-45b4-43f2-b421-cefc07d16204-kube-api-access-fkpgq\") pod \"authorino-6c79785f8f-ggwxc\" (UID: \"0f83ac89-45b4-43f2-b421-cefc07d16204\") " pod="kuadrant-system/authorino-6c79785f8f-ggwxc"
Jul 23 17:14:09.421455 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:14:09.421427    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6c79785f8f-ggwxc"]
Jul 23 17:14:09.421682 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:14:09.421663    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-fkpgq], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-6c79785f8f-ggwxc" podUID="0f83ac89-45b4-43f2-b421-cefc07d16204"
Jul 23 17:14:09.455847 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:14:09.455816    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-75b758965-rtv48"]
Jul 23 17:14:09.459559 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:14:09.459540    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75b758965-rtv48"
Jul 23 17:14:09.461961 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:14:09.461940    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Jul 23 17:14:09.468052 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:14:09.468032    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75b758965-rtv48"]
Jul 23 17:14:09.478852 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:14:09.478828    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkpgq\" (UniqueName: \"kubernetes.io/projected/0f83ac89-45b4-43f2-b421-cefc07d16204-kube-api-access-fkpgq\") pod \"authorino-6c79785f8f-ggwxc\" (UID: \"0f83ac89-45b4-43f2-b421-cefc07d16204\") " pod="kuadrant-system/authorino-6c79785f8f-ggwxc"
Jul 23 17:14:09.491991 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:14:09.491971    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fkpgq\" (UniqueName: \"kubernetes.io/projected/0f83ac89-45b4-43f2-b421-cefc07d16204-kube-api-access-fkpgq\") pod \"authorino-6c79785f8f-ggwxc\" (UID: \"0f83ac89-45b4-43f2-b421-cefc07d16204\") " pod="kuadrant-system/authorino-6c79785f8f-ggwxc"
Jul 23 17:14:09.580195 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:14:09.580107    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/b251316e-9e79-4068-b099-6711e38fb521-tls-cert\") pod \"authorino-75b758965-rtv48\" (UID: \"b251316e-9e79-4068-b099-6711e38fb521\") " pod="kuadrant-system/authorino-75b758965-rtv48"
Jul 23 17:14:09.580195 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:14:09.580178    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqwz5\" (UniqueName: \"kubernetes.io/projected/b251316e-9e79-4068-b099-6711e38fb521-kube-api-access-xqwz5\") pod \"authorino-75b758965-rtv48\" (UID: \"b251316e-9e79-4068-b099-6711e38fb521\") " pod="kuadrant-system/authorino-75b758965-rtv48"
Jul 23 17:14:09.681269 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:14:09.681222    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/b251316e-9e79-4068-b099-6711e38fb521-tls-cert\") pod \"authorino-75b758965-rtv48\" (UID: \"b251316e-9e79-4068-b099-6711e38fb521\") " pod="kuadrant-system/authorino-75b758965-rtv48"
Jul 23 17:14:09.681425 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:14:09.681313    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqwz5\" (UniqueName: \"kubernetes.io/projected/b251316e-9e79-4068-b099-6711e38fb521-kube-api-access-xqwz5\") pod \"authorino-75b758965-rtv48\" (UID: \"b251316e-9e79-4068-b099-6711e38fb521\") " pod="kuadrant-system/authorino-75b758965-rtv48"
Jul 23 17:14:09.683658 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:14:09.683638    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/b251316e-9e79-4068-b099-6711e38fb521-tls-cert\") pod \"authorino-75b758965-rtv48\" (UID: \"b251316e-9e79-4068-b099-6711e38fb521\") " pod="kuadrant-system/authorino-75b758965-rtv48"
Jul 23 17:14:09.689933 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:14:09.689911    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqwz5\" (UniqueName: \"kubernetes.io/projected/b251316e-9e79-4068-b099-6711e38fb521-kube-api-access-xqwz5\") pod \"authorino-75b758965-rtv48\" (UID: \"b251316e-9e79-4068-b099-6711e38fb521\") " pod="kuadrant-system/authorino-75b758965-rtv48"
Jul 23 17:14:09.769676 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:14:09.769642    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75b758965-rtv48"
Jul 23 17:14:09.909111 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:14:09.909085    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75b758965-rtv48"]
Jul 23 17:14:09.911097 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:14:09.911077    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb251316e_9e79_4068_b099_6711e38fb521.slice/crio-eccbee93acfbde031ea3b0401ce22fa1b9659f248ac392b593095d5b57ea4390 WatchSource:0}: Error finding container eccbee93acfbde031ea3b0401ce22fa1b9659f248ac392b593095d5b57ea4390: Status 404 returned error can't find the container with id eccbee93acfbde031ea3b0401ce22fa1b9659f248ac392b593095d5b57ea4390
Jul 23 17:14:10.371161 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:14:10.371124    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-7q22k" event={"ID":"42d626ea-e4bb-439a-83cf-b60dd2f5dff4","Type":"ContainerStarted","Data":"1e91d171f66ea5de9f78cbc8c9c01a2ac1bee2ceb0b024e150fbc1ff1cb21b3a"}
Jul 23 17:14:10.371161 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:14:10.371143    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-5d4b67c446-7q22k" podUID="42d626ea-e4bb-439a-83cf-b60dd2f5dff4" containerName="authorino" containerID="cri-o://1e91d171f66ea5de9f78cbc8c9c01a2ac1bee2ceb0b024e150fbc1ff1cb21b3a" gracePeriod=30
Jul 23 17:14:10.371161 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:14:10.371168    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-7q22k" event={"ID":"42d626ea-e4bb-439a-83cf-b60dd2f5dff4","Type":"ContainerStarted","Data":"1e7ec4cb1ddf61ae1a5205b9a1182d5370c406b60cd512b18255a68db21f8217"}
Jul 23 17:14:10.372447 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:14:10.372428    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6c79785f8f-ggwxc"
Jul 23 17:14:10.372554 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:14:10.372454    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75b758965-rtv48" event={"ID":"b251316e-9e79-4068-b099-6711e38fb521","Type":"ContainerStarted","Data":"eccbee93acfbde031ea3b0401ce22fa1b9659f248ac392b593095d5b57ea4390"}
Jul 23 17:14:10.388346 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:14:10.388305    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5d4b67c446-7q22k" podStartSLOduration=1.9345576960000002 podStartE2EDuration="2.388291208s" totalImagesPullingTime="453.733512ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 17:14:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 17:14:10.385903662 +0000 UTC m=+546.156189182" watchObservedRunningTime="2026-07-23 17:14:10.388291208 +0000 UTC m=+546.158576724"
Jul 23 17:14:10.400167 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:14:10.400143    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6c79785f8f-ggwxc"
Jul 23 17:14:10.486755 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:14:10.486731    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fkpgq\" (UniqueName: \"kubernetes.io/projected/0f83ac89-45b4-43f2-b421-cefc07d16204-kube-api-access-fkpgq\") pod \"0f83ac89-45b4-43f2-b421-cefc07d16204\" (UID: \"0f83ac89-45b4-43f2-b421-cefc07d16204\") "
Jul 23 17:14:10.488527 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:14:10.488501    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f83ac89-45b4-43f2-b421-cefc07d16204-kube-api-access-fkpgq" (OuterVolumeSpecName: "kube-api-access-fkpgq") pod "0f83ac89-45b4-43f2-b421-cefc07d16204" (UID: "0f83ac89-45b4-43f2-b421-cefc07d16204"). InnerVolumeSpecName "kube-api-access-fkpgq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 17:14:10.588208 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:14:10.588180    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fkpgq\" (UniqueName: \"kubernetes.io/projected/0f83ac89-45b4-43f2-b421-cefc07d16204-kube-api-access-fkpgq\") on node \"ip-10-0-129-216.ec2.internal\" DevicePath \"\""
Jul 23 17:14:10.643234 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:14:10.643157    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-7q22k"
Jul 23 17:14:10.790631 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:14:10.790604    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-77nc7\" (UniqueName: \"kubernetes.io/projected/42d626ea-e4bb-439a-83cf-b60dd2f5dff4-kube-api-access-77nc7\") pod \"42d626ea-e4bb-439a-83cf-b60dd2f5dff4\" (UID: \"42d626ea-e4bb-439a-83cf-b60dd2f5dff4\") "
Jul 23 17:14:10.792597 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:14:10.792573    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42d626ea-e4bb-439a-83cf-b60dd2f5dff4-kube-api-access-77nc7" (OuterVolumeSpecName: "kube-api-access-77nc7") pod "42d626ea-e4bb-439a-83cf-b60dd2f5dff4" (UID: "42d626ea-e4bb-439a-83cf-b60dd2f5dff4"). InnerVolumeSpecName "kube-api-access-77nc7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 17:14:10.892102 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:14:10.892071    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-77nc7\" (UniqueName: \"kubernetes.io/projected/42d626ea-e4bb-439a-83cf-b60dd2f5dff4-kube-api-access-77nc7\") on node \"ip-10-0-129-216.ec2.internal\" DevicePath \"\""
Jul 23 17:14:11.378854 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:14:11.378814    2577 generic.go:356] "Generic (PLEG): container finished" podID="42d626ea-e4bb-439a-83cf-b60dd2f5dff4" containerID="1e91d171f66ea5de9f78cbc8c9c01a2ac1bee2ceb0b024e150fbc1ff1cb21b3a" exitCode=0
Jul 23 17:14:11.378854 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:14:11.378857    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-7q22k"
Jul 23 17:14:11.379406 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:14:11.378902    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-7q22k" event={"ID":"42d626ea-e4bb-439a-83cf-b60dd2f5dff4","Type":"ContainerDied","Data":"1e91d171f66ea5de9f78cbc8c9c01a2ac1bee2ceb0b024e150fbc1ff1cb21b3a"}
Jul 23 17:14:11.379406 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:14:11.378946    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-7q22k" event={"ID":"42d626ea-e4bb-439a-83cf-b60dd2f5dff4","Type":"ContainerDied","Data":"1e7ec4cb1ddf61ae1a5205b9a1182d5370c406b60cd512b18255a68db21f8217"}
Jul 23 17:14:11.379406 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:14:11.378969    2577 scope.go:117] "RemoveContainer" containerID="1e91d171f66ea5de9f78cbc8c9c01a2ac1bee2ceb0b024e150fbc1ff1cb21b3a"
Jul 23 17:14:11.380779 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:14:11.380749    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75b758965-rtv48" event={"ID":"b251316e-9e79-4068-b099-6711e38fb521","Type":"ContainerStarted","Data":"cab6eb61800d6e6c5d84a773ad9121eb0c9e3f9cfdffa98bf17d5f22ae37ab33"}
Jul 23 17:14:11.380871 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:14:11.380788    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6c79785f8f-ggwxc"
Jul 23 17:14:11.388332 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:14:11.388318    2577 scope.go:117] "RemoveContainer" containerID="1e91d171f66ea5de9f78cbc8c9c01a2ac1bee2ceb0b024e150fbc1ff1cb21b3a"
Jul 23 17:14:11.388625 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:14:11.388608    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e91d171f66ea5de9f78cbc8c9c01a2ac1bee2ceb0b024e150fbc1ff1cb21b3a\": container with ID starting with 1e91d171f66ea5de9f78cbc8c9c01a2ac1bee2ceb0b024e150fbc1ff1cb21b3a not found: ID does not exist" containerID="1e91d171f66ea5de9f78cbc8c9c01a2ac1bee2ceb0b024e150fbc1ff1cb21b3a"
Jul 23 17:14:11.388693 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:14:11.388636    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e91d171f66ea5de9f78cbc8c9c01a2ac1bee2ceb0b024e150fbc1ff1cb21b3a"} err="failed to get container status \"1e91d171f66ea5de9f78cbc8c9c01a2ac1bee2ceb0b024e150fbc1ff1cb21b3a\": rpc error: code = NotFound desc = could not find container \"1e91d171f66ea5de9f78cbc8c9c01a2ac1bee2ceb0b024e150fbc1ff1cb21b3a\": container with ID starting with 1e91d171f66ea5de9f78cbc8c9c01a2ac1bee2ceb0b024e150fbc1ff1cb21b3a not found: ID does not exist"
Jul 23 17:14:11.404366 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:14:11.404327    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-75b758965-rtv48" podStartSLOduration=1.836261968 podStartE2EDuration="2.404314354s" totalImagesPullingTime="568.052386ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 17:14:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 17:14:11.402838524 +0000 UTC m=+547.173124047" watchObservedRunningTime="2026-07-23 17:14:11.404314354 +0000 UTC m=+547.174599873"
Jul 23 17:14:11.419564 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:14:11.419541    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-7q22k"]
Jul 23 17:14:11.422690 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:14:11.422667    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-7q22k"]
Jul 23 17:14:11.446674 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:14:11.446651    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6c79785f8f-ggwxc"]
Jul 23 17:14:11.452900 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:14:11.452870    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6c79785f8f-ggwxc"]
Jul 23 17:14:12.767452 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:14:12.767416    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0f83ac89-45b4-43f2-b421-cefc07d16204" path="/var/lib/kubelet/pods/0f83ac89-45b4-43f2-b421-cefc07d16204/volumes"
Jul 23 17:14:12.767952 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:14:12.767755    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="42d626ea-e4bb-439a-83cf-b60dd2f5dff4" path="/var/lib/kubelet/pods/42d626ea-e4bb-439a-83cf-b60dd2f5dff4/volumes"
Jul 23 17:14:16.503486 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:14:16.503452    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:14:16.514562 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:14:16.514533    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-577f7b66bd-djd8h"]
Jul 23 17:14:16.514907 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:14:16.514894    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="42d626ea-e4bb-439a-83cf-b60dd2f5dff4" containerName="authorino"
Jul 23 17:14:16.514951 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:14:16.514909    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="42d626ea-e4bb-439a-83cf-b60dd2f5dff4" containerName="authorino"
Jul 23 17:14:16.514996 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:14:16.514987    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="42d626ea-e4bb-439a-83cf-b60dd2f5dff4" containerName="authorino"
Jul 23 17:14:16.534370 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:14:16.534338    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-577f7b66bd-djd8h"]
Jul 23 17:14:16.534536 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:14:16.534435    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-577f7b66bd-djd8h"
Jul 23 17:14:16.537436 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:14:16.537411    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-tz4b7\""
Jul 23 17:14:16.537662 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:14:16.537643    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Jul 23 17:14:16.538330 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:14:16.538313    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Jul 23 17:14:16.589395 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:14:16.589363    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:14:16.643607 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:14:16.643578    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4c552876-ece9-4996-bd66-281a6d2f85c7-webhook-cert\") pod \"maas-controller-577f7b66bd-djd8h\" (UID: \"4c552876-ece9-4996-bd66-281a6d2f85c7\") " pod="opendatahub/maas-controller-577f7b66bd-djd8h"
Jul 23 17:14:16.643786 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:14:16.643636    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxcq8\" (UniqueName: \"kubernetes.io/projected/4c552876-ece9-4996-bd66-281a6d2f85c7-kube-api-access-gxcq8\") pod \"maas-controller-577f7b66bd-djd8h\" (UID: \"4c552876-ece9-4996-bd66-281a6d2f85c7\") " pod="opendatahub/maas-controller-577f7b66bd-djd8h"
Jul 23 17:14:16.653906 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:14:16.653877    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:14:16.699552 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:14:16.699510    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7979b9b8c-xg4sq"]
Jul 23 17:14:16.707009 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:14:16.706981    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7979b9b8c-xg4sq"
Jul 23 17:14:16.713782 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:14:16.713753    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7979b9b8c-xg4sq"]
Jul 23 17:14:16.744519 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:14:16.744492    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4c552876-ece9-4996-bd66-281a6d2f85c7-webhook-cert\") pod \"maas-controller-577f7b66bd-djd8h\" (UID: \"4c552876-ece9-4996-bd66-281a6d2f85c7\") " pod="opendatahub/maas-controller-577f7b66bd-djd8h"
Jul 23 17:14:16.744649 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:14:16.744551    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxcq8\" (UniqueName: \"kubernetes.io/projected/4c552876-ece9-4996-bd66-281a6d2f85c7-kube-api-access-gxcq8\") pod \"maas-controller-577f7b66bd-djd8h\" (UID: \"4c552876-ece9-4996-bd66-281a6d2f85c7\") " pod="opendatahub/maas-controller-577f7b66bd-djd8h"
Jul 23 17:14:16.746996 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:14:16.746968    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4c552876-ece9-4996-bd66-281a6d2f85c7-webhook-cert\") pod \"maas-controller-577f7b66bd-djd8h\" (UID: \"4c552876-ece9-4996-bd66-281a6d2f85c7\") " pod="opendatahub/maas-controller-577f7b66bd-djd8h"
Jul 23 17:14:16.760157 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:14:16.760094    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxcq8\" (UniqueName: \"kubernetes.io/projected/4c552876-ece9-4996-bd66-281a6d2f85c7-kube-api-access-gxcq8\") pod \"maas-controller-577f7b66bd-djd8h\" (UID: \"4c552876-ece9-4996-bd66-281a6d2f85c7\") " pod="opendatahub/maas-controller-577f7b66bd-djd8h"
Jul 23 17:14:16.845699 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:14:16.845672    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/36118a54-8629-421f-851c-a4faf5cfe9e2-webhook-cert\") pod \"maas-controller-7979b9b8c-xg4sq\" (UID: \"36118a54-8629-421f-851c-a4faf5cfe9e2\") " pod="opendatahub/maas-controller-7979b9b8c-xg4sq"
Jul 23 17:14:16.845879 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:14:16.845789    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdblh\" (UniqueName: \"kubernetes.io/projected/36118a54-8629-421f-851c-a4faf5cfe9e2-kube-api-access-rdblh\") pod \"maas-controller-7979b9b8c-xg4sq\" (UID: \"36118a54-8629-421f-851c-a4faf5cfe9e2\") " pod="opendatahub/maas-controller-7979b9b8c-xg4sq"
Jul 23 17:14:16.846134 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:14:16.846119    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-577f7b66bd-djd8h"
Jul 23 17:14:16.947050 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:14:16.947005    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdblh\" (UniqueName: \"kubernetes.io/projected/36118a54-8629-421f-851c-a4faf5cfe9e2-kube-api-access-rdblh\") pod \"maas-controller-7979b9b8c-xg4sq\" (UID: \"36118a54-8629-421f-851c-a4faf5cfe9e2\") " pod="opendatahub/maas-controller-7979b9b8c-xg4sq"
Jul 23 17:14:16.947183 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:14:16.947088    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/36118a54-8629-421f-851c-a4faf5cfe9e2-webhook-cert\") pod \"maas-controller-7979b9b8c-xg4sq\" (UID: \"36118a54-8629-421f-851c-a4faf5cfe9e2\") " pod="opendatahub/maas-controller-7979b9b8c-xg4sq"
Jul 23 17:14:16.949691 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:14:16.949661    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/36118a54-8629-421f-851c-a4faf5cfe9e2-webhook-cert\") pod \"maas-controller-7979b9b8c-xg4sq\" (UID: \"36118a54-8629-421f-851c-a4faf5cfe9e2\") " pod="opendatahub/maas-controller-7979b9b8c-xg4sq"
Jul 23 17:14:16.956400 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:14:16.956375    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdblh\" (UniqueName: \"kubernetes.io/projected/36118a54-8629-421f-851c-a4faf5cfe9e2-kube-api-access-rdblh\") pod \"maas-controller-7979b9b8c-xg4sq\" (UID: \"36118a54-8629-421f-851c-a4faf5cfe9e2\") " pod="opendatahub/maas-controller-7979b9b8c-xg4sq"
Jul 23 17:14:16.976604 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:14:16.976582    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-577f7b66bd-djd8h"]
Jul 23 17:14:16.978220 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:14:16.978185    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c552876_ece9_4996_bd66_281a6d2f85c7.slice/crio-68fd038c07ecb14b8cb8ab27e25eba3ad4755b91b3aca6927695259f57492096 WatchSource:0}: Error finding container 68fd038c07ecb14b8cb8ab27e25eba3ad4755b91b3aca6927695259f57492096: Status 404 returned error can't find the container with id 68fd038c07ecb14b8cb8ab27e25eba3ad4755b91b3aca6927695259f57492096
Jul 23 17:14:17.020217 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:14:17.020161    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7979b9b8c-xg4sq"
Jul 23 17:14:17.152487 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:14:17.152465    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7979b9b8c-xg4sq"]
Jul 23 17:14:17.154499 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:14:17.154473    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36118a54_8629_421f_851c_a4faf5cfe9e2.slice/crio-798e9d45fe0f23b4d3a54ea9b63173045db540af86846eb9b80b50e4381b3917 WatchSource:0}: Error finding container 798e9d45fe0f23b4d3a54ea9b63173045db540af86846eb9b80b50e4381b3917: Status 404 returned error can't find the container with id 798e9d45fe0f23b4d3a54ea9b63173045db540af86846eb9b80b50e4381b3917
Jul 23 17:14:17.290121 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:14:17.290027    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:14:17.412685 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:14:17.412648    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-577f7b66bd-djd8h" event={"ID":"4c552876-ece9-4996-bd66-281a6d2f85c7","Type":"ContainerStarted","Data":"68fd038c07ecb14b8cb8ab27e25eba3ad4755b91b3aca6927695259f57492096"}
Jul 23 17:14:17.413749 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:14:17.413726    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7979b9b8c-xg4sq" event={"ID":"36118a54-8629-421f-851c-a4faf5cfe9e2","Type":"ContainerStarted","Data":"798e9d45fe0f23b4d3a54ea9b63173045db540af86846eb9b80b50e4381b3917"}
Jul 23 17:14:19.671658 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:14:19.671623    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:14:19.777775 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:14:19.777691    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:14:20.442841 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:14:20.442800    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-577f7b66bd-djd8h" event={"ID":"4c552876-ece9-4996-bd66-281a6d2f85c7","Type":"ContainerStarted","Data":"f9c09f6d89502388ff21c9d8d0aca3d1b6bc1ad028bc76df3929a25dfea49da4"}
Jul 23 17:14:20.443085 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:14:20.443010    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-577f7b66bd-djd8h"
Jul 23 17:14:20.444579 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:14:20.444551    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7979b9b8c-xg4sq" event={"ID":"36118a54-8629-421f-851c-a4faf5cfe9e2","Type":"ContainerStarted","Data":"46971ad42e0e93671b172c95e29b06ff47749fa4b3e8bfebc52dcb02dc74ade8"}
Jul 23 17:14:20.444709 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:14:20.444680    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7979b9b8c-xg4sq"
Jul 23 17:14:20.468921 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:14:20.468868    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-577f7b66bd-djd8h" podStartSLOduration=1.946688838 podStartE2EDuration="4.468847286s" totalImagesPullingTime="2.522158448s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 17:14:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 17:14:20.468420164 +0000 UTC m=+556.238705687" watchObservedRunningTime="2026-07-23 17:14:20.468847286 +0000 UTC m=+556.239132810"
Jul 23 17:14:20.491462 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:14:20.491419    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-7979b9b8c-xg4sq" podStartSLOduration=2.154835547 podStartE2EDuration="4.491405092s" totalImagesPullingTime="2.336569545s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 17:14:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 17:14:20.489087909 +0000 UTC m=+556.259373427" watchObservedRunningTime="2026-07-23 17:14:20.491405092 +0000 UTC m=+556.261690611"
Jul 23 17:14:31.452192 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:14:31.452158    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7979b9b8c-xg4sq"
Jul 23 17:14:31.452612 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:14:31.452302    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-577f7b66bd-djd8h"
Jul 23 17:14:31.524507 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:14:31.524476    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-577f7b66bd-djd8h"]
Jul 23 17:14:31.524697 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:14:31.524666    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-577f7b66bd-djd8h" podUID="4c552876-ece9-4996-bd66-281a6d2f85c7" containerName="manager" containerID="cri-o://f9c09f6d89502388ff21c9d8d0aca3d1b6bc1ad028bc76df3929a25dfea49da4" gracePeriod=30
Jul 23 17:14:31.774332 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:14:31.774305    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-577f7b66bd-djd8h"
Jul 23 17:14:31.874476 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:14:31.874445    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4c552876-ece9-4996-bd66-281a6d2f85c7-webhook-cert\") pod \"4c552876-ece9-4996-bd66-281a6d2f85c7\" (UID: \"4c552876-ece9-4996-bd66-281a6d2f85c7\") "
Jul 23 17:14:31.874667 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:14:31.874553    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gxcq8\" (UniqueName: \"kubernetes.io/projected/4c552876-ece9-4996-bd66-281a6d2f85c7-kube-api-access-gxcq8\") pod \"4c552876-ece9-4996-bd66-281a6d2f85c7\" (UID: \"4c552876-ece9-4996-bd66-281a6d2f85c7\") "
Jul 23 17:14:31.876606 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:14:31.876579    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c552876-ece9-4996-bd66-281a6d2f85c7-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "4c552876-ece9-4996-bd66-281a6d2f85c7" (UID: "4c552876-ece9-4996-bd66-281a6d2f85c7"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 17:14:31.876721 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:14:31.876642    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c552876-ece9-4996-bd66-281a6d2f85c7-kube-api-access-gxcq8" (OuterVolumeSpecName: "kube-api-access-gxcq8") pod "4c552876-ece9-4996-bd66-281a6d2f85c7" (UID: "4c552876-ece9-4996-bd66-281a6d2f85c7"). InnerVolumeSpecName "kube-api-access-gxcq8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 17:14:31.975248 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:14:31.975217    2577 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4c552876-ece9-4996-bd66-281a6d2f85c7-webhook-cert\") on node \"ip-10-0-129-216.ec2.internal\" DevicePath \"\""
Jul 23 17:14:31.975248 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:14:31.975243    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gxcq8\" (UniqueName: \"kubernetes.io/projected/4c552876-ece9-4996-bd66-281a6d2f85c7-kube-api-access-gxcq8\") on node \"ip-10-0-129-216.ec2.internal\" DevicePath \"\""
Jul 23 17:14:32.518667 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:14:32.518632    2577 generic.go:356] "Generic (PLEG): container finished" podID="4c552876-ece9-4996-bd66-281a6d2f85c7" containerID="f9c09f6d89502388ff21c9d8d0aca3d1b6bc1ad028bc76df3929a25dfea49da4" exitCode=0
Jul 23 17:14:32.519054 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:14:32.518674    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-577f7b66bd-djd8h"
Jul 23 17:14:32.519054 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:14:32.518722    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-577f7b66bd-djd8h" event={"ID":"4c552876-ece9-4996-bd66-281a6d2f85c7","Type":"ContainerDied","Data":"f9c09f6d89502388ff21c9d8d0aca3d1b6bc1ad028bc76df3929a25dfea49da4"}
Jul 23 17:14:32.519054 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:14:32.518766    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-577f7b66bd-djd8h" event={"ID":"4c552876-ece9-4996-bd66-281a6d2f85c7","Type":"ContainerDied","Data":"68fd038c07ecb14b8cb8ab27e25eba3ad4755b91b3aca6927695259f57492096"}
Jul 23 17:14:32.519054 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:14:32.518785    2577 scope.go:117] "RemoveContainer" containerID="f9c09f6d89502388ff21c9d8d0aca3d1b6bc1ad028bc76df3929a25dfea49da4"
Jul 23 17:14:32.528296 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:14:32.528083    2577 scope.go:117] "RemoveContainer" containerID="f9c09f6d89502388ff21c9d8d0aca3d1b6bc1ad028bc76df3929a25dfea49da4"
Jul 23 17:14:32.528358 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:14:32.528313    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9c09f6d89502388ff21c9d8d0aca3d1b6bc1ad028bc76df3929a25dfea49da4\": container with ID starting with f9c09f6d89502388ff21c9d8d0aca3d1b6bc1ad028bc76df3929a25dfea49da4 not found: ID does not exist" containerID="f9c09f6d89502388ff21c9d8d0aca3d1b6bc1ad028bc76df3929a25dfea49da4"
Jul 23 17:14:32.528358 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:14:32.528337    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9c09f6d89502388ff21c9d8d0aca3d1b6bc1ad028bc76df3929a25dfea49da4"} err="failed to get container status \"f9c09f6d89502388ff21c9d8d0aca3d1b6bc1ad028bc76df3929a25dfea49da4\": rpc error: code = NotFound desc = could not find container \"f9c09f6d89502388ff21c9d8d0aca3d1b6bc1ad028bc76df3929a25dfea49da4\": container with ID starting with f9c09f6d89502388ff21c9d8d0aca3d1b6bc1ad028bc76df3929a25dfea49da4 not found: ID does not exist"
Jul 23 17:14:32.540095 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:14:32.540074    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-577f7b66bd-djd8h"]
Jul 23 17:14:32.544362 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:14:32.544331    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-577f7b66bd-djd8h"]
Jul 23 17:14:32.765437 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:14:32.765405    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4c552876-ece9-4996-bd66-281a6d2f85c7" path="/var/lib/kubelet/pods/4c552876-ece9-4996-bd66-281a6d2f85c7/volumes"
Jul 23 17:14:51.470857 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:14:51.470824    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:14:51.585774 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:14:51.585747    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:14:52.100637 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:14:52.100605    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:14:52.195733 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:14:52.195704    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-7fc9b4c8bc-ng7g6"]
Jul 23 17:14:52.196092 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:14:52.196080    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4c552876-ece9-4996-bd66-281a6d2f85c7" containerName="manager"
Jul 23 17:14:52.196147 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:14:52.196094    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="4c552876-ece9-4996-bd66-281a6d2f85c7" containerName="manager"
Jul 23 17:14:52.196188 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:14:52.196168    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="4c552876-ece9-4996-bd66-281a6d2f85c7" containerName="manager"
Jul 23 17:14:52.201351 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:14:52.201331    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7fc9b4c8bc-ng7g6"
Jul 23 17:14:52.204130 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:14:52.204107    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Jul 23 17:14:52.205306 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:14:52.205280    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-8hgd8\""
Jul 23 17:14:52.209237 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:14:52.209212    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7fc9b4c8bc-ng7g6"]
Jul 23 17:14:52.222491 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:14:52.222466    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:14:52.228678 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:14:52.228658    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/62efdc6c-a1cf-458e-9e9d-190a8130972b-maas-api-tls\") pod \"maas-api-7fc9b4c8bc-ng7g6\" (UID: \"62efdc6c-a1cf-458e-9e9d-190a8130972b\") " pod="odh-ai-gateway-infra/maas-api-7fc9b4c8bc-ng7g6"
Jul 23 17:14:52.228774 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:14:52.228698    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzlnj\" (UniqueName: \"kubernetes.io/projected/62efdc6c-a1cf-458e-9e9d-190a8130972b-kube-api-access-jzlnj\") pod \"maas-api-7fc9b4c8bc-ng7g6\" (UID: \"62efdc6c-a1cf-458e-9e9d-190a8130972b\") " pod="odh-ai-gateway-infra/maas-api-7fc9b4c8bc-ng7g6"
Jul 23 17:14:52.228774 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:14:52.228725    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/62efdc6c-a1cf-458e-9e9d-190a8130972b-openshift-service-ca\") pod \"maas-api-7fc9b4c8bc-ng7g6\" (UID: \"62efdc6c-a1cf-458e-9e9d-190a8130972b\") " pod="odh-ai-gateway-infra/maas-api-7fc9b4c8bc-ng7g6"
Jul 23 17:14:52.329198 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:14:52.329167    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/62efdc6c-a1cf-458e-9e9d-190a8130972b-maas-api-tls\") pod \"maas-api-7fc9b4c8bc-ng7g6\" (UID: \"62efdc6c-a1cf-458e-9e9d-190a8130972b\") " pod="odh-ai-gateway-infra/maas-api-7fc9b4c8bc-ng7g6"
Jul 23 17:14:52.329364 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:14:52.329210    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzlnj\" (UniqueName: \"kubernetes.io/projected/62efdc6c-a1cf-458e-9e9d-190a8130972b-kube-api-access-jzlnj\") pod \"maas-api-7fc9b4c8bc-ng7g6\" (UID: \"62efdc6c-a1cf-458e-9e9d-190a8130972b\") " pod="odh-ai-gateway-infra/maas-api-7fc9b4c8bc-ng7g6"
Jul 23 17:14:52.329364 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:14:52.329232    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/62efdc6c-a1cf-458e-9e9d-190a8130972b-openshift-service-ca\") pod \"maas-api-7fc9b4c8bc-ng7g6\" (UID: \"62efdc6c-a1cf-458e-9e9d-190a8130972b\") " pod="odh-ai-gateway-infra/maas-api-7fc9b4c8bc-ng7g6"
Jul 23 17:14:52.329848 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:14:52.329821    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/62efdc6c-a1cf-458e-9e9d-190a8130972b-openshift-service-ca\") pod \"maas-api-7fc9b4c8bc-ng7g6\" (UID: \"62efdc6c-a1cf-458e-9e9d-190a8130972b\") " pod="odh-ai-gateway-infra/maas-api-7fc9b4c8bc-ng7g6"
Jul 23 17:14:52.331537 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:14:52.331517    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/62efdc6c-a1cf-458e-9e9d-190a8130972b-maas-api-tls\") pod \"maas-api-7fc9b4c8bc-ng7g6\" (UID: \"62efdc6c-a1cf-458e-9e9d-190a8130972b\") " pod="odh-ai-gateway-infra/maas-api-7fc9b4c8bc-ng7g6"
Jul 23 17:14:52.337445 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:14:52.337426    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzlnj\" (UniqueName: \"kubernetes.io/projected/62efdc6c-a1cf-458e-9e9d-190a8130972b-kube-api-access-jzlnj\") pod \"maas-api-7fc9b4c8bc-ng7g6\" (UID: \"62efdc6c-a1cf-458e-9e9d-190a8130972b\") " pod="odh-ai-gateway-infra/maas-api-7fc9b4c8bc-ng7g6"
Jul 23 17:14:52.513198 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:14:52.513168    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7fc9b4c8bc-ng7g6"
Jul 23 17:14:52.675539 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:14:52.675502    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7fc9b4c8bc-ng7g6"]
Jul 23 17:14:52.677458 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:14:52.677423    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:14:52.678486 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:14:52.678462    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62efdc6c_a1cf_458e_9e9d_190a8130972b.slice/crio-58968e797b6cfc49c50af1b47a17d7dff109b3a2104edba4ecc9d587c5a51869 WatchSource:0}: Error finding container 58968e797b6cfc49c50af1b47a17d7dff109b3a2104edba4ecc9d587c5a51869: Status 404 returned error can't find the container with id 58968e797b6cfc49c50af1b47a17d7dff109b3a2104edba4ecc9d587c5a51869
Jul 23 17:14:52.753559 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:14:52.753508    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:14:52.774319 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:14:52.774239    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5bf47cd988-tb75s"]
Jul 23 17:14:52.777496 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:14:52.777477    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5bf47cd988-tb75s"
Jul 23 17:14:52.779894 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:14:52.779876    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 23 17:14:52.780620 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:14:52.780534    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 23 17:14:52.780620 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:14:52.780595    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Jul 23 17:14:52.780794 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:14:52.780686    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-jznzv\""
Jul 23 17:14:52.789488 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:14:52.789467    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5bf47cd988-tb75s"]
Jul 23 17:14:52.834629 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:14:52.834585    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f87b7f7-f222-43da-9f48-37f88ae96fc5-plugins-config-volume\") pod \"payload-processing-5bf47cd988-tb75s\" (UID: \"7f87b7f7-f222-43da-9f48-37f88ae96fc5\") " pod="openshift-ingress/payload-processing-5bf47cd988-tb75s"
Jul 23 17:14:52.834629 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:14:52.834627    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8fghw\" (UniqueName: \"kubernetes.io/projected/7f87b7f7-f222-43da-9f48-37f88ae96fc5-kube-api-access-8fghw\") pod \"payload-processing-5bf47cd988-tb75s\" (UID: \"7f87b7f7-f222-43da-9f48-37f88ae96fc5\") " pod="openshift-ingress/payload-processing-5bf47cd988-tb75s"
Jul 23 17:14:52.936121 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:14:52.936086    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f87b7f7-f222-43da-9f48-37f88ae96fc5-plugins-config-volume\") pod \"payload-processing-5bf47cd988-tb75s\" (UID: \"7f87b7f7-f222-43da-9f48-37f88ae96fc5\") " pod="openshift-ingress/payload-processing-5bf47cd988-tb75s"
Jul 23 17:14:52.936121 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:14:52.936120    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8fghw\" (UniqueName: \"kubernetes.io/projected/7f87b7f7-f222-43da-9f48-37f88ae96fc5-kube-api-access-8fghw\") pod \"payload-processing-5bf47cd988-tb75s\" (UID: \"7f87b7f7-f222-43da-9f48-37f88ae96fc5\") " pod="openshift-ingress/payload-processing-5bf47cd988-tb75s"
Jul 23 17:14:52.936790 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:14:52.936770    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f87b7f7-f222-43da-9f48-37f88ae96fc5-plugins-config-volume\") pod \"payload-processing-5bf47cd988-tb75s\" (UID: \"7f87b7f7-f222-43da-9f48-37f88ae96fc5\") " pod="openshift-ingress/payload-processing-5bf47cd988-tb75s"
Jul 23 17:14:52.944227 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:14:52.944200    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8fghw\" (UniqueName: \"kubernetes.io/projected/7f87b7f7-f222-43da-9f48-37f88ae96fc5-kube-api-access-8fghw\") pod \"payload-processing-5bf47cd988-tb75s\" (UID: \"7f87b7f7-f222-43da-9f48-37f88ae96fc5\") " pod="openshift-ingress/payload-processing-5bf47cd988-tb75s"
Jul 23 17:14:53.089362 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:14:53.089288    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5bf47cd988-tb75s"
Jul 23 17:14:53.226957 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:14:53.226935    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5bf47cd988-tb75s"]
Jul 23 17:14:53.228174 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:14:53.228147    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f87b7f7_f222_43da_9f48_37f88ae96fc5.slice/crio-d0943565c69f476e3d4eeffd3c1a257e1fb2a5af04831ce22150ca5ec37832f9 WatchSource:0}: Error finding container d0943565c69f476e3d4eeffd3c1a257e1fb2a5af04831ce22150ca5ec37832f9: Status 404 returned error can't find the container with id d0943565c69f476e3d4eeffd3c1a257e1fb2a5af04831ce22150ca5ec37832f9
Jul 23 17:14:53.634605 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:14:53.634563    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7fc9b4c8bc-ng7g6" event={"ID":"62efdc6c-a1cf-458e-9e9d-190a8130972b","Type":"ContainerStarted","Data":"58968e797b6cfc49c50af1b47a17d7dff109b3a2104edba4ecc9d587c5a51869"}
Jul 23 17:14:53.635717 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:14:53.635690    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5bf47cd988-tb75s" event={"ID":"7f87b7f7-f222-43da-9f48-37f88ae96fc5","Type":"ContainerStarted","Data":"d0943565c69f476e3d4eeffd3c1a257e1fb2a5af04831ce22150ca5ec37832f9"}
Jul 23 17:14:55.649766 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:14:55.649704    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7fc9b4c8bc-ng7g6" event={"ID":"62efdc6c-a1cf-458e-9e9d-190a8130972b","Type":"ContainerStarted","Data":"98bc63ad0b63b224935dd49fbac08e8a4f102b942beda67b48a212cf62dbc050"}
Jul 23 17:14:55.650285 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:14:55.649803    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-7fc9b4c8bc-ng7g6"
Jul 23 17:14:55.651438 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:14:55.651411    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5bf47cd988-tb75s" event={"ID":"7f87b7f7-f222-43da-9f48-37f88ae96fc5","Type":"ContainerStarted","Data":"96a8ee3cabf9caf02a9a9f14436a031ccc2c8d6f0a860136954d1a1468f6863b"}
Jul 23 17:14:55.651560 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:14:55.651503    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5bf47cd988-tb75s"
Jul 23 17:14:55.668882 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:14:55.668834    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-7fc9b4c8bc-ng7g6" podStartSLOduration=1.260228811 podStartE2EDuration="3.668819428s" totalImagesPullingTime="2.408590617s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 17:14:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 17:14:55.666686356 +0000 UTC m=+591.436971876" watchObservedRunningTime="2026-07-23 17:14:55.668819428 +0000 UTC m=+591.439104946"
Jul 23 17:14:55.683814 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:14:55.683776    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5bf47cd988-tb75s" podStartSLOduration=1.822918357 podStartE2EDuration="3.683762538s" totalImagesPullingTime="1.860844181s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 17:14:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 17:14:55.681322466 +0000 UTC m=+591.451607989" watchObservedRunningTime="2026-07-23 17:14:55.683762538 +0000 UTC m=+591.454048056"
Jul 23 17:15:01.663056 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:15:01.663024    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-7fc9b4c8bc-ng7g6"
Jul 23 17:15:02.207869 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:15:02.207821    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:15:04.350002 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:15:04.349953    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:15:06.660436 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:15:06.660406    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5bf47cd988-tb75s"
Jul 23 17:15:09.829145 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:15:09.829109    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:15:22.536356 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:15:22.536326    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:15:25.105100 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:15:25.105041    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-5ffcv"]
Jul 23 17:15:25.107788 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:15:25.107769    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5ffcv"
Jul 23 17:15:25.111203 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:15:25.111177    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-4mqfb\""
Jul 23 17:15:25.111345 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:15:25.111217    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\""
Jul 23 17:15:25.111345 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:15:25.111216    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Jul 23 17:15:25.111345 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:15:25.111277    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Jul 23 17:15:25.120730 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:15:25.120706    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-5ffcv"]
Jul 23 17:15:25.133157 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:15:25.133126    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76383eb1-5f14-4225-9233-deac178d77ba-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-5ffcv\" (UID: \"76383eb1-5f14-4225-9233-deac178d77ba\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5ffcv"
Jul 23 17:15:25.133315 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:15:25.133184    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76383eb1-5f14-4225-9233-deac178d77ba-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-5ffcv\" (UID: \"76383eb1-5f14-4225-9233-deac178d77ba\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5ffcv"
Jul 23 17:15:25.133315 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:15:25.133212    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76383eb1-5f14-4225-9233-deac178d77ba-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-5ffcv\" (UID: \"76383eb1-5f14-4225-9233-deac178d77ba\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5ffcv"
Jul 23 17:15:25.133315 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:15:25.133270    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76383eb1-5f14-4225-9233-deac178d77ba-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-5ffcv\" (UID: \"76383eb1-5f14-4225-9233-deac178d77ba\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5ffcv"
Jul 23 17:15:25.133496 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:15:25.133318    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7h7z\" (UniqueName: \"kubernetes.io/projected/76383eb1-5f14-4225-9233-deac178d77ba-kube-api-access-x7h7z\") pod \"e2e-trlp-test-simulated-kserve-598466548d-5ffcv\" (UID: \"76383eb1-5f14-4225-9233-deac178d77ba\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5ffcv"
Jul 23 17:15:25.133496 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:15:25.133390    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76383eb1-5f14-4225-9233-deac178d77ba-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-5ffcv\" (UID: \"76383eb1-5f14-4225-9233-deac178d77ba\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5ffcv"
Jul 23 17:15:25.143216 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:15:25.143190    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:15:25.233985 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:15:25.233941    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76383eb1-5f14-4225-9233-deac178d77ba-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-5ffcv\" (UID: \"76383eb1-5f14-4225-9233-deac178d77ba\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5ffcv"
Jul 23 17:15:25.233985 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:15:25.233986    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76383eb1-5f14-4225-9233-deac178d77ba-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-5ffcv\" (UID: \"76383eb1-5f14-4225-9233-deac178d77ba\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5ffcv"
Jul 23 17:15:25.234283 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:15:25.234040    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76383eb1-5f14-4225-9233-deac178d77ba-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-5ffcv\" (UID: \"76383eb1-5f14-4225-9233-deac178d77ba\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5ffcv"
Jul 23 17:15:25.234283 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:15:25.234091    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7h7z\" (UniqueName: \"kubernetes.io/projected/76383eb1-5f14-4225-9233-deac178d77ba-kube-api-access-x7h7z\") pod \"e2e-trlp-test-simulated-kserve-598466548d-5ffcv\" (UID: \"76383eb1-5f14-4225-9233-deac178d77ba\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5ffcv"
Jul 23 17:15:25.234283 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:15:25.234144    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76383eb1-5f14-4225-9233-deac178d77ba-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-5ffcv\" (UID: \"76383eb1-5f14-4225-9233-deac178d77ba\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5ffcv"
Jul 23 17:15:25.234283 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:15:25.234199    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76383eb1-5f14-4225-9233-deac178d77ba-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-5ffcv\" (UID: \"76383eb1-5f14-4225-9233-deac178d77ba\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5ffcv"
Jul 23 17:15:25.234527 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:15:25.234425    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76383eb1-5f14-4225-9233-deac178d77ba-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-5ffcv\" (UID: \"76383eb1-5f14-4225-9233-deac178d77ba\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5ffcv"
Jul 23 17:15:25.234590 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:15:25.234531    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76383eb1-5f14-4225-9233-deac178d77ba-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-5ffcv\" (UID: \"76383eb1-5f14-4225-9233-deac178d77ba\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5ffcv"
Jul 23 17:15:25.234590 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:15:25.234537    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76383eb1-5f14-4225-9233-deac178d77ba-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-5ffcv\" (UID: \"76383eb1-5f14-4225-9233-deac178d77ba\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5ffcv"
Jul 23 17:15:25.236429 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:15:25.236405    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76383eb1-5f14-4225-9233-deac178d77ba-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-5ffcv\" (UID: \"76383eb1-5f14-4225-9233-deac178d77ba\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5ffcv"
Jul 23 17:15:25.236627 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:15:25.236610    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76383eb1-5f14-4225-9233-deac178d77ba-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-5ffcv\" (UID: \"76383eb1-5f14-4225-9233-deac178d77ba\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5ffcv"
Jul 23 17:15:25.245794 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:15:25.245761    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7h7z\" (UniqueName: \"kubernetes.io/projected/76383eb1-5f14-4225-9233-deac178d77ba-kube-api-access-x7h7z\") pod \"e2e-trlp-test-simulated-kserve-598466548d-5ffcv\" (UID: \"76383eb1-5f14-4225-9233-deac178d77ba\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5ffcv"
Jul 23 17:15:25.419858 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:15:25.419780    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5ffcv"
Jul 23 17:15:25.564548 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:15:25.564524    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-5ffcv"]
Jul 23 17:15:25.567232 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:15:25.567201    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76383eb1_5f14_4225_9233_deac178d77ba.slice/crio-207844445f29bde054979d4fe16ce9777c1c949193ad780e720d12e4b1a93bfd WatchSource:0}: Error finding container 207844445f29bde054979d4fe16ce9777c1c949193ad780e720d12e4b1a93bfd: Status 404 returned error can't find the container with id 207844445f29bde054979d4fe16ce9777c1c949193ad780e720d12e4b1a93bfd
Jul 23 17:15:25.568713 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:15:25.568692    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 23 17:15:25.813550 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:15:25.813517    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5ffcv" event={"ID":"76383eb1-5f14-4225-9233-deac178d77ba","Type":"ContainerStarted","Data":"207844445f29bde054979d4fe16ce9777c1c949193ad780e720d12e4b1a93bfd"}
Jul 23 17:15:26.820972 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:15:26.820938    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5ffcv" event={"ID":"76383eb1-5f14-4225-9233-deac178d77ba","Type":"ContainerStarted","Data":"a8e8ce5e2a07d8e8576454f8b8e7fe639c033f3f36478d536aeffaf32c965c6b"}
Jul 23 17:15:26.844440 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:15:26.844388    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5ffcv" podStartSLOduration=0.669370279 podStartE2EDuration="1.84437138s" totalImagesPullingTime="1.175001101s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 17:15:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 17:15:26.840795501 +0000 UTC m=+622.611081021" watchObservedRunningTime="2026-07-23 17:15:26.84437138 +0000 UTC m=+622.614656921"
Jul 23 17:15:35.420601 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:15:35.420562    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5ffcv"
Jul 23 17:15:35.421122 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:15:35.420627    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5ffcv"
Jul 23 17:15:35.434031 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:15:35.434006    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5ffcv"
Jul 23 17:15:35.880631 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:15:35.880597    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5ffcv"
Jul 23 17:15:44.616843 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:15:44.616805    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:15:58.400878 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:15:58.400845    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-msdd5"]
Jul 23 17:15:58.404624 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:15:58.404590    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-msdd5"
Jul 23 17:15:58.407737 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:15:58.407715    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\""
Jul 23 17:15:58.415603 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:15:58.415580    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-msdd5"]
Jul 23 17:15:58.451473 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:15:58.451430    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:15:58.525929 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:15:58.525900    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/02ac9ef4-9ce4-49b5-8a1b-4b337b4ed85e-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-msdd5\" (UID: \"02ac9ef4-9ce4-49b5-8a1b-4b337b4ed85e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-msdd5"
Jul 23 17:15:58.526075 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:15:58.525995    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mfm2j\" (UniqueName: \"kubernetes.io/projected/02ac9ef4-9ce4-49b5-8a1b-4b337b4ed85e-kube-api-access-mfm2j\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-msdd5\" (UID: \"02ac9ef4-9ce4-49b5-8a1b-4b337b4ed85e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-msdd5"
Jul 23 17:15:58.526075 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:15:58.526031    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/02ac9ef4-9ce4-49b5-8a1b-4b337b4ed85e-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-msdd5\" (UID: \"02ac9ef4-9ce4-49b5-8a1b-4b337b4ed85e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-msdd5"
Jul 23 17:15:58.526075 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:15:58.526068    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/02ac9ef4-9ce4-49b5-8a1b-4b337b4ed85e-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-msdd5\" (UID: \"02ac9ef4-9ce4-49b5-8a1b-4b337b4ed85e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-msdd5"
Jul 23 17:15:58.526199 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:15:58.526098    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/02ac9ef4-9ce4-49b5-8a1b-4b337b4ed85e-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-msdd5\" (UID: \"02ac9ef4-9ce4-49b5-8a1b-4b337b4ed85e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-msdd5"
Jul 23 17:15:58.526199 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:15:58.526140    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/02ac9ef4-9ce4-49b5-8a1b-4b337b4ed85e-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-msdd5\" (UID: \"02ac9ef4-9ce4-49b5-8a1b-4b337b4ed85e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-msdd5"
Jul 23 17:15:58.627589 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:15:58.627552    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/02ac9ef4-9ce4-49b5-8a1b-4b337b4ed85e-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-msdd5\" (UID: \"02ac9ef4-9ce4-49b5-8a1b-4b337b4ed85e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-msdd5"
Jul 23 17:15:58.627750 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:15:58.627643    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/02ac9ef4-9ce4-49b5-8a1b-4b337b4ed85e-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-msdd5\" (UID: \"02ac9ef4-9ce4-49b5-8a1b-4b337b4ed85e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-msdd5"
Jul 23 17:15:58.627750 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:15:58.627692    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/02ac9ef4-9ce4-49b5-8a1b-4b337b4ed85e-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-msdd5\" (UID: \"02ac9ef4-9ce4-49b5-8a1b-4b337b4ed85e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-msdd5"
Jul 23 17:15:58.627844 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:15:58.627796    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mfm2j\" (UniqueName: \"kubernetes.io/projected/02ac9ef4-9ce4-49b5-8a1b-4b337b4ed85e-kube-api-access-mfm2j\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-msdd5\" (UID: \"02ac9ef4-9ce4-49b5-8a1b-4b337b4ed85e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-msdd5"
Jul 23 17:15:58.627844 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:15:58.627830    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/02ac9ef4-9ce4-49b5-8a1b-4b337b4ed85e-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-msdd5\" (UID: \"02ac9ef4-9ce4-49b5-8a1b-4b337b4ed85e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-msdd5"
Jul 23 17:15:58.627956 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:15:58.627888    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/02ac9ef4-9ce4-49b5-8a1b-4b337b4ed85e-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-msdd5\" (UID: \"02ac9ef4-9ce4-49b5-8a1b-4b337b4ed85e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-msdd5"
Jul 23 17:15:58.628007 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:15:58.627961    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/02ac9ef4-9ce4-49b5-8a1b-4b337b4ed85e-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-msdd5\" (UID: \"02ac9ef4-9ce4-49b5-8a1b-4b337b4ed85e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-msdd5"
Jul 23 17:15:58.628062 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:15:58.628008    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/02ac9ef4-9ce4-49b5-8a1b-4b337b4ed85e-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-msdd5\" (UID: \"02ac9ef4-9ce4-49b5-8a1b-4b337b4ed85e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-msdd5"
Jul 23 17:15:58.628146 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:15:58.628128    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/02ac9ef4-9ce4-49b5-8a1b-4b337b4ed85e-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-msdd5\" (UID: \"02ac9ef4-9ce4-49b5-8a1b-4b337b4ed85e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-msdd5"
Jul 23 17:15:58.629866 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:15:58.629835    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/02ac9ef4-9ce4-49b5-8a1b-4b337b4ed85e-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-msdd5\" (UID: \"02ac9ef4-9ce4-49b5-8a1b-4b337b4ed85e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-msdd5"
Jul 23 17:15:58.630043 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:15:58.630025    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/02ac9ef4-9ce4-49b5-8a1b-4b337b4ed85e-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-msdd5\" (UID: \"02ac9ef4-9ce4-49b5-8a1b-4b337b4ed85e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-msdd5"
Jul 23 17:15:58.637583 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:15:58.637556    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mfm2j\" (UniqueName: \"kubernetes.io/projected/02ac9ef4-9ce4-49b5-8a1b-4b337b4ed85e-kube-api-access-mfm2j\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-msdd5\" (UID: \"02ac9ef4-9ce4-49b5-8a1b-4b337b4ed85e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-msdd5"
Jul 23 17:15:58.716359 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:15:58.716323    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-msdd5"
Jul 23 17:15:58.868695 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:15:58.868669    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-msdd5"]
Jul 23 17:15:58.870930 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:15:58.870905    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02ac9ef4_9ce4_49b5_8a1b_4b337b4ed85e.slice/crio-22f00a9f1c36b699db355d81103ded18bc06a620784ddb5fe0913746e4bf6fc0 WatchSource:0}: Error finding container 22f00a9f1c36b699db355d81103ded18bc06a620784ddb5fe0913746e4bf6fc0: Status 404 returned error can't find the container with id 22f00a9f1c36b699db355d81103ded18bc06a620784ddb5fe0913746e4bf6fc0
Jul 23 17:15:58.999911 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:15:58.999816    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-msdd5" event={"ID":"02ac9ef4-9ce4-49b5-8a1b-4b337b4ed85e","Type":"ContainerStarted","Data":"22f00a9f1c36b699db355d81103ded18bc06a620784ddb5fe0913746e4bf6fc0"}
Jul 23 17:16:01.013464 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:16:01.013429    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-msdd5" event={"ID":"02ac9ef4-9ce4-49b5-8a1b-4b337b4ed85e","Type":"ContainerStarted","Data":"6b98b438b8ba830c310fcf033986c37d7ca55076a502dda56c941787e71fe81e"}
Jul 23 17:16:01.036428 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:16:01.036377    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-msdd5" podStartSLOduration=1.295805225 podStartE2EDuration="3.036359991s" totalImagesPullingTime="1.740554766s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 17:15:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 17:16:01.032567495 +0000 UTC m=+656.802853016" watchObservedRunningTime="2026-07-23 17:16:01.036359991 +0000 UTC m=+656.806645552"
Jul 23 17:16:08.486971 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:16:08.486930    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:16:08.716792 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:16:08.716753    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-msdd5"
Jul 23 17:16:08.716957 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:16:08.716820    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-msdd5"
Jul 23 17:16:08.729124 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:16:08.729094    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-msdd5"
Jul 23 17:16:09.075224 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:16:09.075197    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-msdd5"
Jul 23 17:16:39.672048 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:16:39.672018    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:16:39.734671 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:16:39.734622    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:17:13.082375 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:17:13.082316    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:17:13.179704 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:17:13.179672    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:17:13.828787 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:17:13.828753    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75b758965-rtv48"]
Jul 23 17:17:13.829022 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:17:13.828984    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-75b758965-rtv48" podUID="b251316e-9e79-4068-b099-6711e38fb521" containerName="authorino" containerID="cri-o://cab6eb61800d6e6c5d84a773ad9121eb0c9e3f9cfdffa98bf17d5f22ae37ab33" gracePeriod=30
Jul 23 17:17:14.091675 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:17:14.091615    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75b758965-rtv48"
Jul 23 17:17:14.190849 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:17:14.190815    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/b251316e-9e79-4068-b099-6711e38fb521-tls-cert\") pod \"b251316e-9e79-4068-b099-6711e38fb521\" (UID: \"b251316e-9e79-4068-b099-6711e38fb521\") "
Jul 23 17:17:14.191015 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:17:14.190917    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xqwz5\" (UniqueName: \"kubernetes.io/projected/b251316e-9e79-4068-b099-6711e38fb521-kube-api-access-xqwz5\") pod \"b251316e-9e79-4068-b099-6711e38fb521\" (UID: \"b251316e-9e79-4068-b099-6711e38fb521\") "
Jul 23 17:17:14.193121 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:17:14.193084    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b251316e-9e79-4068-b099-6711e38fb521-kube-api-access-xqwz5" (OuterVolumeSpecName: "kube-api-access-xqwz5") pod "b251316e-9e79-4068-b099-6711e38fb521" (UID: "b251316e-9e79-4068-b099-6711e38fb521"). InnerVolumeSpecName "kube-api-access-xqwz5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 17:17:14.201997 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:17:14.201972    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b251316e-9e79-4068-b099-6711e38fb521-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "b251316e-9e79-4068-b099-6711e38fb521" (UID: "b251316e-9e79-4068-b099-6711e38fb521"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 17:17:14.258736 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:17:14.258679    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:17:14.292226 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:17:14.292194    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xqwz5\" (UniqueName: \"kubernetes.io/projected/b251316e-9e79-4068-b099-6711e38fb521-kube-api-access-xqwz5\") on node \"ip-10-0-129-216.ec2.internal\" DevicePath \"\""
Jul 23 17:17:14.292226 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:17:14.292221    2577 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/b251316e-9e79-4068-b099-6711e38fb521-tls-cert\") on node \"ip-10-0-129-216.ec2.internal\" DevicePath \"\""
Jul 23 17:17:14.385971 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:17:14.385874    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:17:14.422842 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:17:14.422799    2577 generic.go:356] "Generic (PLEG): container finished" podID="b251316e-9e79-4068-b099-6711e38fb521" containerID="cab6eb61800d6e6c5d84a773ad9121eb0c9e3f9cfdffa98bf17d5f22ae37ab33" exitCode=0
Jul 23 17:17:14.422842 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:17:14.422844    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75b758965-rtv48"
Jul 23 17:17:14.423066 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:17:14.422883    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75b758965-rtv48" event={"ID":"b251316e-9e79-4068-b099-6711e38fb521","Type":"ContainerDied","Data":"cab6eb61800d6e6c5d84a773ad9121eb0c9e3f9cfdffa98bf17d5f22ae37ab33"}
Jul 23 17:17:14.423066 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:17:14.422930    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75b758965-rtv48" event={"ID":"b251316e-9e79-4068-b099-6711e38fb521","Type":"ContainerDied","Data":"eccbee93acfbde031ea3b0401ce22fa1b9659f248ac392b593095d5b57ea4390"}
Jul 23 17:17:14.423066 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:17:14.422951    2577 scope.go:117] "RemoveContainer" containerID="cab6eb61800d6e6c5d84a773ad9121eb0c9e3f9cfdffa98bf17d5f22ae37ab33"
Jul 23 17:17:14.432930 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:17:14.432912    2577 scope.go:117] "RemoveContainer" containerID="cab6eb61800d6e6c5d84a773ad9121eb0c9e3f9cfdffa98bf17d5f22ae37ab33"
Jul 23 17:17:14.433222 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:17:14.433202    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cab6eb61800d6e6c5d84a773ad9121eb0c9e3f9cfdffa98bf17d5f22ae37ab33\": container with ID starting with cab6eb61800d6e6c5d84a773ad9121eb0c9e3f9cfdffa98bf17d5f22ae37ab33 not found: ID does not exist" containerID="cab6eb61800d6e6c5d84a773ad9121eb0c9e3f9cfdffa98bf17d5f22ae37ab33"
Jul 23 17:17:14.433364 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:17:14.433233    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cab6eb61800d6e6c5d84a773ad9121eb0c9e3f9cfdffa98bf17d5f22ae37ab33"} err="failed to get container status \"cab6eb61800d6e6c5d84a773ad9121eb0c9e3f9cfdffa98bf17d5f22ae37ab33\": rpc error: code = NotFound desc = could not find container \"cab6eb61800d6e6c5d84a773ad9121eb0c9e3f9cfdffa98bf17d5f22ae37ab33\": container with ID starting with cab6eb61800d6e6c5d84a773ad9121eb0c9e3f9cfdffa98bf17d5f22ae37ab33 not found: ID does not exist"
Jul 23 17:17:14.471370 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:17:14.471337    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75b758965-rtv48"]
Jul 23 17:17:14.485911 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:17:14.485878    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-75b758965-rtv48"]
Jul 23 17:17:14.766952 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:17:14.766918    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b251316e-9e79-4068-b099-6711e38fb521" path="/var/lib/kubelet/pods/b251316e-9e79-4068-b099-6711e38fb521/volumes"
Jul 23 17:17:17.503544 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:17:17.503509    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:17:17.688353 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:17:17.688323    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:17:55.657589 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:17:55.657548    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:18:01.400348 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:18:01.400302    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7979b9b8c-xg4sq"]
Jul 23 17:18:01.400804 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:18:01.400579    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-7979b9b8c-xg4sq" podUID="36118a54-8629-421f-851c-a4faf5cfe9e2" containerName="manager" containerID="cri-o://46971ad42e0e93671b172c95e29b06ff47749fa4b3e8bfebc52dcb02dc74ade8" gracePeriod=30
Jul 23 17:18:01.451302 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:18:01.451262    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="opendatahub/maas-controller-7979b9b8c-xg4sq" podUID="36118a54-8629-421f-851c-a4faf5cfe9e2" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.47:8081/readyz\": dial tcp 10.133.0.47:8081: connect: connection refused"
Jul 23 17:18:01.649865 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:18:01.649840    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7979b9b8c-xg4sq"
Jul 23 17:18:01.687646 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:18:01.687554    2577 generic.go:356] "Generic (PLEG): container finished" podID="36118a54-8629-421f-851c-a4faf5cfe9e2" containerID="46971ad42e0e93671b172c95e29b06ff47749fa4b3e8bfebc52dcb02dc74ade8" exitCode=0
Jul 23 17:18:01.687646 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:18:01.687607    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7979b9b8c-xg4sq"
Jul 23 17:18:01.687878 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:18:01.687643    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7979b9b8c-xg4sq" event={"ID":"36118a54-8629-421f-851c-a4faf5cfe9e2","Type":"ContainerDied","Data":"46971ad42e0e93671b172c95e29b06ff47749fa4b3e8bfebc52dcb02dc74ade8"}
Jul 23 17:18:01.687878 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:18:01.687710    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7979b9b8c-xg4sq" event={"ID":"36118a54-8629-421f-851c-a4faf5cfe9e2","Type":"ContainerDied","Data":"798e9d45fe0f23b4d3a54ea9b63173045db540af86846eb9b80b50e4381b3917"}
Jul 23 17:18:01.687878 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:18:01.687779    2577 scope.go:117] "RemoveContainer" containerID="46971ad42e0e93671b172c95e29b06ff47749fa4b3e8bfebc52dcb02dc74ade8"
Jul 23 17:18:01.697761 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:18:01.697738    2577 scope.go:117] "RemoveContainer" containerID="46971ad42e0e93671b172c95e29b06ff47749fa4b3e8bfebc52dcb02dc74ade8"
Jul 23 17:18:01.698020 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:18:01.698001    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46971ad42e0e93671b172c95e29b06ff47749fa4b3e8bfebc52dcb02dc74ade8\": container with ID starting with 46971ad42e0e93671b172c95e29b06ff47749fa4b3e8bfebc52dcb02dc74ade8 not found: ID does not exist" containerID="46971ad42e0e93671b172c95e29b06ff47749fa4b3e8bfebc52dcb02dc74ade8"
Jul 23 17:18:01.698111 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:18:01.698029    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46971ad42e0e93671b172c95e29b06ff47749fa4b3e8bfebc52dcb02dc74ade8"} err="failed to get container status \"46971ad42e0e93671b172c95e29b06ff47749fa4b3e8bfebc52dcb02dc74ade8\": rpc error: code = NotFound desc = could not find container \"46971ad42e0e93671b172c95e29b06ff47749fa4b3e8bfebc52dcb02dc74ade8\": container with ID starting with 46971ad42e0e93671b172c95e29b06ff47749fa4b3e8bfebc52dcb02dc74ade8 not found: ID does not exist"
Jul 23 17:18:01.722569 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:18:01.722536    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rdblh\" (UniqueName: \"kubernetes.io/projected/36118a54-8629-421f-851c-a4faf5cfe9e2-kube-api-access-rdblh\") pod \"36118a54-8629-421f-851c-a4faf5cfe9e2\" (UID: \"36118a54-8629-421f-851c-a4faf5cfe9e2\") "
Jul 23 17:18:01.722692 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:18:01.722641    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/36118a54-8629-421f-851c-a4faf5cfe9e2-webhook-cert\") pod \"36118a54-8629-421f-851c-a4faf5cfe9e2\" (UID: \"36118a54-8629-421f-851c-a4faf5cfe9e2\") "
Jul 23 17:18:01.724565 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:18:01.724539    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/36118a54-8629-421f-851c-a4faf5cfe9e2-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "36118a54-8629-421f-851c-a4faf5cfe9e2" (UID: "36118a54-8629-421f-851c-a4faf5cfe9e2"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 17:18:01.724962 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:18:01.724932    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36118a54-8629-421f-851c-a4faf5cfe9e2-kube-api-access-rdblh" (OuterVolumeSpecName: "kube-api-access-rdblh") pod "36118a54-8629-421f-851c-a4faf5cfe9e2" (UID: "36118a54-8629-421f-851c-a4faf5cfe9e2"). InnerVolumeSpecName "kube-api-access-rdblh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 17:18:01.823850 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:18:01.823810    2577 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/36118a54-8629-421f-851c-a4faf5cfe9e2-webhook-cert\") on node \"ip-10-0-129-216.ec2.internal\" DevicePath \"\""
Jul 23 17:18:01.823850 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:18:01.823844    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rdblh\" (UniqueName: \"kubernetes.io/projected/36118a54-8629-421f-851c-a4faf5cfe9e2-kube-api-access-rdblh\") on node \"ip-10-0-129-216.ec2.internal\" DevicePath \"\""
Jul 23 17:18:02.010173 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:18:02.010143    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7979b9b8c-xg4sq"]
Jul 23 17:18:02.014438 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:18:02.014411    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-7979b9b8c-xg4sq"]
Jul 23 17:18:02.767593 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:18:02.767559    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="36118a54-8629-421f-851c-a4faf5cfe9e2" path="/var/lib/kubelet/pods/36118a54-8629-421f-851c-a4faf5cfe9e2/volumes"
Jul 23 17:18:28.053957 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:18:28.053881    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:19:02.848964 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:19:02.848925    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7979b9b8c-67cx9"]
Jul 23 17:19:02.849479 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:19:02.849355    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b251316e-9e79-4068-b099-6711e38fb521" containerName="authorino"
Jul 23 17:19:02.849479 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:19:02.849367    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="b251316e-9e79-4068-b099-6711e38fb521" containerName="authorino"
Jul 23 17:19:02.849479 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:19:02.849379    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="36118a54-8629-421f-851c-a4faf5cfe9e2" containerName="manager"
Jul 23 17:19:02.849479 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:19:02.849385    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="36118a54-8629-421f-851c-a4faf5cfe9e2" containerName="manager"
Jul 23 17:19:02.849479 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:19:02.849467    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="b251316e-9e79-4068-b099-6711e38fb521" containerName="authorino"
Jul 23 17:19:02.849479 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:19:02.849475    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="36118a54-8629-421f-851c-a4faf5cfe9e2" containerName="manager"
Jul 23 17:19:02.852716 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:19:02.852698    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7979b9b8c-67cx9"
Jul 23 17:19:02.856146 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:19:02.856124    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Jul 23 17:19:02.856302 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:19:02.856126    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-tz4b7\""
Jul 23 17:19:02.856302 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:19:02.856152    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Jul 23 17:19:02.863235 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:19:02.863213    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7979b9b8c-67cx9"]
Jul 23 17:19:02.976272 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:19:02.976219    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29z4h\" (UniqueName: \"kubernetes.io/projected/ebbc9207-ec43-411f-ae7b-10c9c5bbb81a-kube-api-access-29z4h\") pod \"maas-controller-7979b9b8c-67cx9\" (UID: \"ebbc9207-ec43-411f-ae7b-10c9c5bbb81a\") " pod="opendatahub/maas-controller-7979b9b8c-67cx9"
Jul 23 17:19:02.976441 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:19:02.976394    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ebbc9207-ec43-411f-ae7b-10c9c5bbb81a-webhook-cert\") pod \"maas-controller-7979b9b8c-67cx9\" (UID: \"ebbc9207-ec43-411f-ae7b-10c9c5bbb81a\") " pod="opendatahub/maas-controller-7979b9b8c-67cx9"
Jul 23 17:19:03.077532 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:19:03.077498    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ebbc9207-ec43-411f-ae7b-10c9c5bbb81a-webhook-cert\") pod \"maas-controller-7979b9b8c-67cx9\" (UID: \"ebbc9207-ec43-411f-ae7b-10c9c5bbb81a\") " pod="opendatahub/maas-controller-7979b9b8c-67cx9"
Jul 23 17:19:03.077735 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:19:03.077546    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29z4h\" (UniqueName: \"kubernetes.io/projected/ebbc9207-ec43-411f-ae7b-10c9c5bbb81a-kube-api-access-29z4h\") pod \"maas-controller-7979b9b8c-67cx9\" (UID: \"ebbc9207-ec43-411f-ae7b-10c9c5bbb81a\") " pod="opendatahub/maas-controller-7979b9b8c-67cx9"
Jul 23 17:19:03.079943 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:19:03.079919    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ebbc9207-ec43-411f-ae7b-10c9c5bbb81a-webhook-cert\") pod \"maas-controller-7979b9b8c-67cx9\" (UID: \"ebbc9207-ec43-411f-ae7b-10c9c5bbb81a\") " pod="opendatahub/maas-controller-7979b9b8c-67cx9"
Jul 23 17:19:03.085958 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:19:03.085939    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-29z4h\" (UniqueName: \"kubernetes.io/projected/ebbc9207-ec43-411f-ae7b-10c9c5bbb81a-kube-api-access-29z4h\") pod \"maas-controller-7979b9b8c-67cx9\" (UID: \"ebbc9207-ec43-411f-ae7b-10c9c5bbb81a\") " pod="opendatahub/maas-controller-7979b9b8c-67cx9"
Jul 23 17:19:03.162989 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:19:03.162912    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7979b9b8c-67cx9"
Jul 23 17:19:03.300779 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:19:03.300755    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7979b9b8c-67cx9"]
Jul 23 17:19:03.302415 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:19:03.302380    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebbc9207_ec43_411f_ae7b_10c9c5bbb81a.slice/crio-2da3a9e327e0cfdf6da4329cd2d6b6e71b84bdba041853d4af2a8fefe50ca822 WatchSource:0}: Error finding container 2da3a9e327e0cfdf6da4329cd2d6b6e71b84bdba041853d4af2a8fefe50ca822: Status 404 returned error can't find the container with id 2da3a9e327e0cfdf6da4329cd2d6b6e71b84bdba041853d4af2a8fefe50ca822
Jul 23 17:19:04.019687 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:19:04.019650    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7979b9b8c-67cx9" event={"ID":"ebbc9207-ec43-411f-ae7b-10c9c5bbb81a","Type":"ContainerStarted","Data":"ff611a5f698e1c0582dc162745d3995323996bdbeba25259da495444dba4ed58"}
Jul 23 17:19:04.020169 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:19:04.019712    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7979b9b8c-67cx9" event={"ID":"ebbc9207-ec43-411f-ae7b-10c9c5bbb81a","Type":"ContainerStarted","Data":"2da3a9e327e0cfdf6da4329cd2d6b6e71b84bdba041853d4af2a8fefe50ca822"}
Jul 23 17:19:04.020169 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:19:04.019772    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7979b9b8c-67cx9"
Jul 23 17:19:04.063024 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:19:04.062910    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-7979b9b8c-67cx9" podStartSLOduration=1.59036599 podStartE2EDuration="2.062893763s" totalImagesPullingTime="472.527773ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 17:19:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 17:19:04.061603153 +0000 UTC m=+839.831888684" watchObservedRunningTime="2026-07-23 17:19:04.062893763 +0000 UTC m=+839.833179282"
Jul 23 17:19:15.030645 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:19:15.030613    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7979b9b8c-67cx9"
Jul 23 17:19:19.599022 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:19:19.598983    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:19:19.696729 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:19:19.696693    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7979b9b8c-67cx9"]
Jul 23 17:19:19.696907 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:19:19.696885    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-7979b9b8c-67cx9" podUID="ebbc9207-ec43-411f-ae7b-10c9c5bbb81a" containerName="manager" containerID="cri-o://ff611a5f698e1c0582dc162745d3995323996bdbeba25259da495444dba4ed58" gracePeriod=30
Jul 23 17:19:19.947339 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:19:19.947314    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7979b9b8c-67cx9"
Jul 23 17:19:20.024946 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:19:20.024912    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-29z4h\" (UniqueName: \"kubernetes.io/projected/ebbc9207-ec43-411f-ae7b-10c9c5bbb81a-kube-api-access-29z4h\") pod \"ebbc9207-ec43-411f-ae7b-10c9c5bbb81a\" (UID: \"ebbc9207-ec43-411f-ae7b-10c9c5bbb81a\") "
Jul 23 17:19:20.025107 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:19:20.024996    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ebbc9207-ec43-411f-ae7b-10c9c5bbb81a-webhook-cert\") pod \"ebbc9207-ec43-411f-ae7b-10c9c5bbb81a\" (UID: \"ebbc9207-ec43-411f-ae7b-10c9c5bbb81a\") "
Jul 23 17:19:20.026919 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:19:20.026891    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebbc9207-ec43-411f-ae7b-10c9c5bbb81a-kube-api-access-29z4h" (OuterVolumeSpecName: "kube-api-access-29z4h") pod "ebbc9207-ec43-411f-ae7b-10c9c5bbb81a" (UID: "ebbc9207-ec43-411f-ae7b-10c9c5bbb81a"). InnerVolumeSpecName "kube-api-access-29z4h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 17:19:20.026919 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:19:20.026911    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ebbc9207-ec43-411f-ae7b-10c9c5bbb81a-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "ebbc9207-ec43-411f-ae7b-10c9c5bbb81a" (UID: "ebbc9207-ec43-411f-ae7b-10c9c5bbb81a"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 17:19:20.113019 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:19:20.112987    2577 generic.go:356] "Generic (PLEG): container finished" podID="ebbc9207-ec43-411f-ae7b-10c9c5bbb81a" containerID="ff611a5f698e1c0582dc162745d3995323996bdbeba25259da495444dba4ed58" exitCode=0
Jul 23 17:19:20.113179 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:19:20.113032    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7979b9b8c-67cx9"
Jul 23 17:19:20.113179 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:19:20.113044    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7979b9b8c-67cx9" event={"ID":"ebbc9207-ec43-411f-ae7b-10c9c5bbb81a","Type":"ContainerDied","Data":"ff611a5f698e1c0582dc162745d3995323996bdbeba25259da495444dba4ed58"}
Jul 23 17:19:20.113179 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:19:20.113087    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7979b9b8c-67cx9" event={"ID":"ebbc9207-ec43-411f-ae7b-10c9c5bbb81a","Type":"ContainerDied","Data":"2da3a9e327e0cfdf6da4329cd2d6b6e71b84bdba041853d4af2a8fefe50ca822"}
Jul 23 17:19:20.113179 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:19:20.113107    2577 scope.go:117] "RemoveContainer" containerID="ff611a5f698e1c0582dc162745d3995323996bdbeba25259da495444dba4ed58"
Jul 23 17:19:20.122156 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:19:20.122138    2577 scope.go:117] "RemoveContainer" containerID="ff611a5f698e1c0582dc162745d3995323996bdbeba25259da495444dba4ed58"
Jul 23 17:19:20.122450 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:19:20.122429    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff611a5f698e1c0582dc162745d3995323996bdbeba25259da495444dba4ed58\": container with ID starting with ff611a5f698e1c0582dc162745d3995323996bdbeba25259da495444dba4ed58 not found: ID does not exist" containerID="ff611a5f698e1c0582dc162745d3995323996bdbeba25259da495444dba4ed58"
Jul 23 17:19:20.122539 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:19:20.122456    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff611a5f698e1c0582dc162745d3995323996bdbeba25259da495444dba4ed58"} err="failed to get container status \"ff611a5f698e1c0582dc162745d3995323996bdbeba25259da495444dba4ed58\": rpc error: code = NotFound desc = could not find container \"ff611a5f698e1c0582dc162745d3995323996bdbeba25259da495444dba4ed58\": container with ID starting with ff611a5f698e1c0582dc162745d3995323996bdbeba25259da495444dba4ed58 not found: ID does not exist"
Jul 23 17:19:20.125607 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:19:20.125587    2577 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ebbc9207-ec43-411f-ae7b-10c9c5bbb81a-webhook-cert\") on node \"ip-10-0-129-216.ec2.internal\" DevicePath \"\""
Jul 23 17:19:20.125693 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:19:20.125610    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-29z4h\" (UniqueName: \"kubernetes.io/projected/ebbc9207-ec43-411f-ae7b-10c9c5bbb81a-kube-api-access-29z4h\") on node \"ip-10-0-129-216.ec2.internal\" DevicePath \"\""
Jul 23 17:19:20.136307 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:19:20.136285    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7979b9b8c-67cx9"]
Jul 23 17:19:20.139755 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:19:20.139735    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-7979b9b8c-67cx9"]
Jul 23 17:19:20.766765 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:19:20.766735    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ebbc9207-ec43-411f-ae7b-10c9c5bbb81a" path="/var/lib/kubelet/pods/ebbc9207-ec43-411f-ae7b-10c9c5bbb81a/volumes"
Jul 23 17:19:20.812058 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:19:20.812026    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:19:20.916228 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:19:20.916192    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7979b9b8c-vm92w"]
Jul 23 17:19:20.916688 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:19:20.916669    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ebbc9207-ec43-411f-ae7b-10c9c5bbb81a" containerName="manager"
Jul 23 17:19:20.916772 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:19:20.916692    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="ebbc9207-ec43-411f-ae7b-10c9c5bbb81a" containerName="manager"
Jul 23 17:19:20.916832 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:19:20.916817    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="ebbc9207-ec43-411f-ae7b-10c9c5bbb81a" containerName="manager"
Jul 23 17:19:20.921442 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:19:20.921421    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7979b9b8c-vm92w"
Jul 23 17:19:20.923888 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:19:20.923870    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Jul 23 17:19:20.924804 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:19:20.924776    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-tz4b7\""
Jul 23 17:19:20.924918 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:19:20.924796    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Jul 23 17:19:20.927895 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:19:20.927874    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7979b9b8c-vm92w"]
Jul 23 17:19:20.929842 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:19:20.929819    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3f763d30-b0dc-4dd9-b0b1-b91ee5c631f9-webhook-cert\") pod \"maas-controller-7979b9b8c-vm92w\" (UID: \"3f763d30-b0dc-4dd9-b0b1-b91ee5c631f9\") " pod="opendatahub/maas-controller-7979b9b8c-vm92w"
Jul 23 17:19:20.929954 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:19:20.929855    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dr6jr\" (UniqueName: \"kubernetes.io/projected/3f763d30-b0dc-4dd9-b0b1-b91ee5c631f9-kube-api-access-dr6jr\") pod \"maas-controller-7979b9b8c-vm92w\" (UID: \"3f763d30-b0dc-4dd9-b0b1-b91ee5c631f9\") " pod="opendatahub/maas-controller-7979b9b8c-vm92w"
Jul 23 17:19:21.030994 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:19:21.030904    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3f763d30-b0dc-4dd9-b0b1-b91ee5c631f9-webhook-cert\") pod \"maas-controller-7979b9b8c-vm92w\" (UID: \"3f763d30-b0dc-4dd9-b0b1-b91ee5c631f9\") " pod="opendatahub/maas-controller-7979b9b8c-vm92w"
Jul 23 17:19:21.030994 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:19:21.030952    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dr6jr\" (UniqueName: \"kubernetes.io/projected/3f763d30-b0dc-4dd9-b0b1-b91ee5c631f9-kube-api-access-dr6jr\") pod \"maas-controller-7979b9b8c-vm92w\" (UID: \"3f763d30-b0dc-4dd9-b0b1-b91ee5c631f9\") " pod="opendatahub/maas-controller-7979b9b8c-vm92w"
Jul 23 17:19:21.033394 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:19:21.033373    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3f763d30-b0dc-4dd9-b0b1-b91ee5c631f9-webhook-cert\") pod \"maas-controller-7979b9b8c-vm92w\" (UID: \"3f763d30-b0dc-4dd9-b0b1-b91ee5c631f9\") " pod="opendatahub/maas-controller-7979b9b8c-vm92w"
Jul 23 17:19:21.040629 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:19:21.040605    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dr6jr\" (UniqueName: \"kubernetes.io/projected/3f763d30-b0dc-4dd9-b0b1-b91ee5c631f9-kube-api-access-dr6jr\") pod \"maas-controller-7979b9b8c-vm92w\" (UID: \"3f763d30-b0dc-4dd9-b0b1-b91ee5c631f9\") " pod="opendatahub/maas-controller-7979b9b8c-vm92w"
Jul 23 17:19:21.233301 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:19:21.233230    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7979b9b8c-vm92w"
Jul 23 17:19:21.368371 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:19:21.365124    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7979b9b8c-vm92w"]
Jul 23 17:19:22.125465 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:19:22.125372    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7979b9b8c-vm92w" event={"ID":"3f763d30-b0dc-4dd9-b0b1-b91ee5c631f9","Type":"ContainerStarted","Data":"5fe521f9e2e4d571a92e2dd4db13567c51b089fbf73ac3970257ef87f8a1dd4c"}
Jul 23 17:19:22.125465 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:19:22.125440    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7979b9b8c-vm92w"
Jul 23 17:19:22.125465 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:19:22.125460    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7979b9b8c-vm92w" event={"ID":"3f763d30-b0dc-4dd9-b0b1-b91ee5c631f9","Type":"ContainerStarted","Data":"d1ba8cdedaf161c59d849c20667377a13b5cc93b506776638b4083d2d43773f0"}
Jul 23 17:19:22.145815 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:19:22.145764    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-7979b9b8c-vm92w" podStartSLOduration=1.74180435 podStartE2EDuration="2.145750327s" totalImagesPullingTime="403.945977ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 17:19:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 17:19:22.143089565 +0000 UTC m=+857.913375083" watchObservedRunningTime="2026-07-23 17:19:22.145750327 +0000 UTC m=+857.916035845"
Jul 23 17:19:22.318895 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:19:22.318866    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:19:23.460989 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:19:23.460950    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:19:33.133038 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:19:33.133007    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7979b9b8c-vm92w"
Jul 23 17:20:01.830453 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:20:01.830370    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:20:01.931558 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:20:01.931523    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:20:02.075933 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:20:02.075899    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:20:10.647185 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:20:10.647141    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:20:12.001850 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:20:12.001817    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:20:17.507388 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:20:17.507349    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:20:17.769721 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:20:17.769627    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:20:38.760829 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:20:38.760787    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:20:38.861415 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:20:38.861377    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:24:18.360498 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:24:18.360468    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mgqbt"]
Jul 23 17:24:18.361020 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:24:18.360685    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mgqbt" podUID="a695e55b-cb43-455b-9eec-ddca2b0c6f99" containerName="manager" containerID="cri-o://8ef36b1cad084b860a163955bd3806b49cd59e3c4166d5d85a887bd26f672ffa" gracePeriod=10
Jul 23 17:24:21.308512 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:24:21.308487    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mgqbt"
Jul 23 17:24:21.450286 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:24:21.450237    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s9dw7\" (UniqueName: \"kubernetes.io/projected/a695e55b-cb43-455b-9eec-ddca2b0c6f99-kube-api-access-s9dw7\") pod \"a695e55b-cb43-455b-9eec-ddca2b0c6f99\" (UID: \"a695e55b-cb43-455b-9eec-ddca2b0c6f99\") "
Jul 23 17:24:21.450461 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:24:21.450353    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a695e55b-cb43-455b-9eec-ddca2b0c6f99-extensions-socket-volume\") pod \"a695e55b-cb43-455b-9eec-ddca2b0c6f99\" (UID: \"a695e55b-cb43-455b-9eec-ddca2b0c6f99\") "
Jul 23 17:24:21.450742 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:24:21.450716    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a695e55b-cb43-455b-9eec-ddca2b0c6f99-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "a695e55b-cb43-455b-9eec-ddca2b0c6f99" (UID: "a695e55b-cb43-455b-9eec-ddca2b0c6f99"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 17:24:21.452272 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:24:21.452233    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a695e55b-cb43-455b-9eec-ddca2b0c6f99-kube-api-access-s9dw7" (OuterVolumeSpecName: "kube-api-access-s9dw7") pod "a695e55b-cb43-455b-9eec-ddca2b0c6f99" (UID: "a695e55b-cb43-455b-9eec-ddca2b0c6f99"). InnerVolumeSpecName "kube-api-access-s9dw7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 17:24:21.551691 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:24:21.551662    2577 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a695e55b-cb43-455b-9eec-ddca2b0c6f99-extensions-socket-volume\") on node \"ip-10-0-129-216.ec2.internal\" DevicePath \"\""
Jul 23 17:24:21.551691 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:24:21.551686    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s9dw7\" (UniqueName: \"kubernetes.io/projected/a695e55b-cb43-455b-9eec-ddca2b0c6f99-kube-api-access-s9dw7\") on node \"ip-10-0-129-216.ec2.internal\" DevicePath \"\""
Jul 23 17:24:21.796018 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:24:21.795933    2577 generic.go:356] "Generic (PLEG): container finished" podID="a695e55b-cb43-455b-9eec-ddca2b0c6f99" containerID="8ef36b1cad084b860a163955bd3806b49cd59e3c4166d5d85a887bd26f672ffa" exitCode=0
Jul 23 17:24:21.796018 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:24:21.795982    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mgqbt"
Jul 23 17:24:21.796196 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:24:21.796015    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mgqbt" event={"ID":"a695e55b-cb43-455b-9eec-ddca2b0c6f99","Type":"ContainerDied","Data":"8ef36b1cad084b860a163955bd3806b49cd59e3c4166d5d85a887bd26f672ffa"}
Jul 23 17:24:21.796196 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:24:21.796068    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mgqbt" event={"ID":"a695e55b-cb43-455b-9eec-ddca2b0c6f99","Type":"ContainerDied","Data":"3c8d57a040db065b1d87ba96c2db577e9aa37db2eef9d0e424a2f3b926c3525a"}
Jul 23 17:24:21.796196 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:24:21.796090    2577 scope.go:117] "RemoveContainer" containerID="8ef36b1cad084b860a163955bd3806b49cd59e3c4166d5d85a887bd26f672ffa"
Jul 23 17:24:21.806582 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:24:21.806563    2577 scope.go:117] "RemoveContainer" containerID="8ef36b1cad084b860a163955bd3806b49cd59e3c4166d5d85a887bd26f672ffa"
Jul 23 17:24:21.806828 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:24:21.806809    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ef36b1cad084b860a163955bd3806b49cd59e3c4166d5d85a887bd26f672ffa\": container with ID starting with 8ef36b1cad084b860a163955bd3806b49cd59e3c4166d5d85a887bd26f672ffa not found: ID does not exist" containerID="8ef36b1cad084b860a163955bd3806b49cd59e3c4166d5d85a887bd26f672ffa"
Jul 23 17:24:21.806891 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:24:21.806838    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ef36b1cad084b860a163955bd3806b49cd59e3c4166d5d85a887bd26f672ffa"} err="failed to get container status \"8ef36b1cad084b860a163955bd3806b49cd59e3c4166d5d85a887bd26f672ffa\": rpc error: code = NotFound desc = could not find container \"8ef36b1cad084b860a163955bd3806b49cd59e3c4166d5d85a887bd26f672ffa\": container with ID starting with 8ef36b1cad084b860a163955bd3806b49cd59e3c4166d5d85a887bd26f672ffa not found: ID does not exist"
Jul 23 17:24:21.819659 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:24:21.819635    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mgqbt"]
Jul 23 17:24:21.825072 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:24:21.825050    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mgqbt"]
Jul 23 17:24:22.766055 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:24:22.766017    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a695e55b-cb43-455b-9eec-ddca2b0c6f99" path="/var/lib/kubelet/pods/a695e55b-cb43-455b-9eec-ddca2b0c6f99/volumes"
Jul 23 17:25:19.553115 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:25:19.553078    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hxc6t"]
Jul 23 17:25:19.555430 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:25:19.553513    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a695e55b-cb43-455b-9eec-ddca2b0c6f99" containerName="manager"
Jul 23 17:25:19.555430 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:25:19.553526    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="a695e55b-cb43-455b-9eec-ddca2b0c6f99" containerName="manager"
Jul 23 17:25:19.555430 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:25:19.553607    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="a695e55b-cb43-455b-9eec-ddca2b0c6f99" containerName="manager"
Jul 23 17:25:19.556732 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:25:19.556717    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hxc6t"
Jul 23 17:25:19.559229 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:25:19.559203    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-pdv8c\""
Jul 23 17:25:19.572847 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:25:19.572823    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hxc6t"]
Jul 23 17:25:19.663980 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:25:19.663948    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wfn98\" (UniqueName: \"kubernetes.io/projected/dbd9fff4-9aef-4496-b026-e7cc838cd76f-kube-api-access-wfn98\") pod \"kuadrant-operator-controller-manager-86878c769f-hxc6t\" (UID: \"dbd9fff4-9aef-4496-b026-e7cc838cd76f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hxc6t"
Jul 23 17:25:19.663980 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:25:19.663982    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/dbd9fff4-9aef-4496-b026-e7cc838cd76f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-hxc6t\" (UID: \"dbd9fff4-9aef-4496-b026-e7cc838cd76f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hxc6t"
Jul 23 17:25:19.765007 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:25:19.764973    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wfn98\" (UniqueName: \"kubernetes.io/projected/dbd9fff4-9aef-4496-b026-e7cc838cd76f-kube-api-access-wfn98\") pod \"kuadrant-operator-controller-manager-86878c769f-hxc6t\" (UID: \"dbd9fff4-9aef-4496-b026-e7cc838cd76f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hxc6t"
Jul 23 17:25:19.765007 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:25:19.765005    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/dbd9fff4-9aef-4496-b026-e7cc838cd76f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-hxc6t\" (UID: \"dbd9fff4-9aef-4496-b026-e7cc838cd76f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hxc6t"
Jul 23 17:25:19.765367 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:25:19.765351    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/dbd9fff4-9aef-4496-b026-e7cc838cd76f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-hxc6t\" (UID: \"dbd9fff4-9aef-4496-b026-e7cc838cd76f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hxc6t"
Jul 23 17:25:19.774093 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:25:19.774072    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wfn98\" (UniqueName: \"kubernetes.io/projected/dbd9fff4-9aef-4496-b026-e7cc838cd76f-kube-api-access-wfn98\") pod \"kuadrant-operator-controller-manager-86878c769f-hxc6t\" (UID: \"dbd9fff4-9aef-4496-b026-e7cc838cd76f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hxc6t"
Jul 23 17:25:19.866621 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:25:19.866521    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hxc6t"
Jul 23 17:25:20.010407 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:25:20.010376    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hxc6t"]
Jul 23 17:25:20.011039 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:25:20.011010    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbd9fff4_9aef_4496_b026_e7cc838cd76f.slice/crio-47bcd2fff7426199fbb08e7913aad235b43fd574a2c39e439b896f171e662f29 WatchSource:0}: Error finding container 47bcd2fff7426199fbb08e7913aad235b43fd574a2c39e439b896f171e662f29: Status 404 returned error can't find the container with id 47bcd2fff7426199fbb08e7913aad235b43fd574a2c39e439b896f171e662f29
Jul 23 17:25:20.110131 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:25:20.110100    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hxc6t" event={"ID":"dbd9fff4-9aef-4496-b026-e7cc838cd76f","Type":"ContainerStarted","Data":"516a27ffd87a48d836cc26cb27f90fdaefde68eed112f175a716a48724931672"}
Jul 23 17:25:20.110288 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:25:20.110148    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hxc6t" event={"ID":"dbd9fff4-9aef-4496-b026-e7cc838cd76f","Type":"ContainerStarted","Data":"47bcd2fff7426199fbb08e7913aad235b43fd574a2c39e439b896f171e662f29"}
Jul 23 17:25:20.110288 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:25:20.110206    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hxc6t"
Jul 23 17:25:20.169607 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:25:20.169513    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hxc6t" podStartSLOduration=1.169497131 podStartE2EDuration="1.169497131s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 17:25:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 17:25:20.168988235 +0000 UTC m=+1215.939273764" watchObservedRunningTime="2026-07-23 17:25:20.169497131 +0000 UTC m=+1215.939782649"
Jul 23 17:25:31.118154 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:25:31.118123    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hxc6t"
Jul 23 17:26:28.013386 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:26:28.013302    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:26:30.168268 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:26:30.168228    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:27:57.424398 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:27:57.424363    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:28:16.512614 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:28:16.512581    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-3f08f93f-kserve-f56c5c8f5-gvb5q"]
Jul 23 17:28:16.515929 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:28:16.515913    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-3f08f93f-kserve-f56c5c8f5-gvb5q"
Jul 23 17:28:16.518526 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:28:16.518505    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-3f08f93f-kserve-self-signed-certs\""
Jul 23 17:28:16.535043 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:28:16.535020    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-3f08f93f-kserve-f56c5c8f5-gvb5q"]
Jul 23 17:28:16.612438 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:28:16.612393    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:28:16.623607 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:28:16.623576    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d7a13c10-a53e-42e5-875a-abb1e88ab69c-dshm\") pod \"e2e-same-id-a-3f08f93f-kserve-f56c5c8f5-gvb5q\" (UID: \"d7a13c10-a53e-42e5-875a-abb1e88ab69c\") " pod="llm/e2e-same-id-a-3f08f93f-kserve-f56c5c8f5-gvb5q"
Jul 23 17:28:16.623748 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:28:16.623635    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d7a13c10-a53e-42e5-875a-abb1e88ab69c-tls-certs\") pod \"e2e-same-id-a-3f08f93f-kserve-f56c5c8f5-gvb5q\" (UID: \"d7a13c10-a53e-42e5-875a-abb1e88ab69c\") " pod="llm/e2e-same-id-a-3f08f93f-kserve-f56c5c8f5-gvb5q"
Jul 23 17:28:16.623748 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:28:16.623671    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d7a13c10-a53e-42e5-875a-abb1e88ab69c-model-cache\") pod \"e2e-same-id-a-3f08f93f-kserve-f56c5c8f5-gvb5q\" (UID: \"d7a13c10-a53e-42e5-875a-abb1e88ab69c\") " pod="llm/e2e-same-id-a-3f08f93f-kserve-f56c5c8f5-gvb5q"
Jul 23 17:28:16.623748 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:28:16.623715    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d7a13c10-a53e-42e5-875a-abb1e88ab69c-home\") pod \"e2e-same-id-a-3f08f93f-kserve-f56c5c8f5-gvb5q\" (UID: \"d7a13c10-a53e-42e5-875a-abb1e88ab69c\") " pod="llm/e2e-same-id-a-3f08f93f-kserve-f56c5c8f5-gvb5q"
Jul 23 17:28:16.623921 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:28:16.623764    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xzw4z\" (UniqueName: \"kubernetes.io/projected/d7a13c10-a53e-42e5-875a-abb1e88ab69c-kube-api-access-xzw4z\") pod \"e2e-same-id-a-3f08f93f-kserve-f56c5c8f5-gvb5q\" (UID: \"d7a13c10-a53e-42e5-875a-abb1e88ab69c\") " pod="llm/e2e-same-id-a-3f08f93f-kserve-f56c5c8f5-gvb5q"
Jul 23 17:28:16.623921 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:28:16.623818    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d7a13c10-a53e-42e5-875a-abb1e88ab69c-tmp-dir\") pod \"e2e-same-id-a-3f08f93f-kserve-f56c5c8f5-gvb5q\" (UID: \"d7a13c10-a53e-42e5-875a-abb1e88ab69c\") " pod="llm/e2e-same-id-a-3f08f93f-kserve-f56c5c8f5-gvb5q"
Jul 23 17:28:16.725155 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:28:16.725122    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d7a13c10-a53e-42e5-875a-abb1e88ab69c-tls-certs\") pod \"e2e-same-id-a-3f08f93f-kserve-f56c5c8f5-gvb5q\" (UID: \"d7a13c10-a53e-42e5-875a-abb1e88ab69c\") " pod="llm/e2e-same-id-a-3f08f93f-kserve-f56c5c8f5-gvb5q"
Jul 23 17:28:16.725358 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:28:16.725170    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d7a13c10-a53e-42e5-875a-abb1e88ab69c-model-cache\") pod \"e2e-same-id-a-3f08f93f-kserve-f56c5c8f5-gvb5q\" (UID: \"d7a13c10-a53e-42e5-875a-abb1e88ab69c\") " pod="llm/e2e-same-id-a-3f08f93f-kserve-f56c5c8f5-gvb5q"
Jul 23 17:28:16.725358 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:28:16.725205    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d7a13c10-a53e-42e5-875a-abb1e88ab69c-home\") pod \"e2e-same-id-a-3f08f93f-kserve-f56c5c8f5-gvb5q\" (UID: \"d7a13c10-a53e-42e5-875a-abb1e88ab69c\") " pod="llm/e2e-same-id-a-3f08f93f-kserve-f56c5c8f5-gvb5q"
Jul 23 17:28:16.725358 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:28:16.725230    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzw4z\" (UniqueName: \"kubernetes.io/projected/d7a13c10-a53e-42e5-875a-abb1e88ab69c-kube-api-access-xzw4z\") pod \"e2e-same-id-a-3f08f93f-kserve-f56c5c8f5-gvb5q\" (UID: \"d7a13c10-a53e-42e5-875a-abb1e88ab69c\") " pod="llm/e2e-same-id-a-3f08f93f-kserve-f56c5c8f5-gvb5q"
Jul 23 17:28:16.725358 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:28:16.725313    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d7a13c10-a53e-42e5-875a-abb1e88ab69c-tmp-dir\") pod \"e2e-same-id-a-3f08f93f-kserve-f56c5c8f5-gvb5q\" (UID: \"d7a13c10-a53e-42e5-875a-abb1e88ab69c\") " pod="llm/e2e-same-id-a-3f08f93f-kserve-f56c5c8f5-gvb5q"
Jul 23 17:28:16.725605 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:28:16.725383    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d7a13c10-a53e-42e5-875a-abb1e88ab69c-dshm\") pod \"e2e-same-id-a-3f08f93f-kserve-f56c5c8f5-gvb5q\" (UID: \"d7a13c10-a53e-42e5-875a-abb1e88ab69c\") " pod="llm/e2e-same-id-a-3f08f93f-kserve-f56c5c8f5-gvb5q"
Jul 23 17:28:16.725709 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:28:16.725679    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d7a13c10-a53e-42e5-875a-abb1e88ab69c-model-cache\") pod \"e2e-same-id-a-3f08f93f-kserve-f56c5c8f5-gvb5q\" (UID: \"d7a13c10-a53e-42e5-875a-abb1e88ab69c\") " pod="llm/e2e-same-id-a-3f08f93f-kserve-f56c5c8f5-gvb5q"
Jul 23 17:28:16.725773 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:28:16.725689    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d7a13c10-a53e-42e5-875a-abb1e88ab69c-home\") pod \"e2e-same-id-a-3f08f93f-kserve-f56c5c8f5-gvb5q\" (UID: \"d7a13c10-a53e-42e5-875a-abb1e88ab69c\") " pod="llm/e2e-same-id-a-3f08f93f-kserve-f56c5c8f5-gvb5q"
Jul 23 17:28:16.725823 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:28:16.725781    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d7a13c10-a53e-42e5-875a-abb1e88ab69c-tmp-dir\") pod \"e2e-same-id-a-3f08f93f-kserve-f56c5c8f5-gvb5q\" (UID: \"d7a13c10-a53e-42e5-875a-abb1e88ab69c\") " pod="llm/e2e-same-id-a-3f08f93f-kserve-f56c5c8f5-gvb5q"
Jul 23 17:28:16.727570 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:28:16.727549    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d7a13c10-a53e-42e5-875a-abb1e88ab69c-dshm\") pod \"e2e-same-id-a-3f08f93f-kserve-f56c5c8f5-gvb5q\" (UID: \"d7a13c10-a53e-42e5-875a-abb1e88ab69c\") " pod="llm/e2e-same-id-a-3f08f93f-kserve-f56c5c8f5-gvb5q"
Jul 23 17:28:16.727746 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:28:16.727729    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d7a13c10-a53e-42e5-875a-abb1e88ab69c-tls-certs\") pod \"e2e-same-id-a-3f08f93f-kserve-f56c5c8f5-gvb5q\" (UID: \"d7a13c10-a53e-42e5-875a-abb1e88ab69c\") " pod="llm/e2e-same-id-a-3f08f93f-kserve-f56c5c8f5-gvb5q"
Jul 23 17:28:16.737271 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:28:16.737238    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xzw4z\" (UniqueName: \"kubernetes.io/projected/d7a13c10-a53e-42e5-875a-abb1e88ab69c-kube-api-access-xzw4z\") pod \"e2e-same-id-a-3f08f93f-kserve-f56c5c8f5-gvb5q\" (UID: \"d7a13c10-a53e-42e5-875a-abb1e88ab69c\") " pod="llm/e2e-same-id-a-3f08f93f-kserve-f56c5c8f5-gvb5q"
Jul 23 17:28:16.824814 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:28:16.824738    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-3f08f93f-kserve-f56c5c8f5-gvb5q"
Jul 23 17:28:16.974080 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:28:16.974052    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-3f08f93f-kserve-f56c5c8f5-gvb5q"]
Jul 23 17:28:16.975490 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:28:16.975460    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd7a13c10_a53e_42e5_875a_abb1e88ab69c.slice/crio-c1fec1f41149c986696d6a266d1a58eeea2ba242f4be4d2a3da9d9cec40246df WatchSource:0}: Error finding container c1fec1f41149c986696d6a266d1a58eeea2ba242f4be4d2a3da9d9cec40246df: Status 404 returned error can't find the container with id c1fec1f41149c986696d6a266d1a58eeea2ba242f4be4d2a3da9d9cec40246df
Jul 23 17:28:17.082993 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:28:17.082904    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-3f08f93f-kserve-f56c5c8f5-gvb5q" event={"ID":"d7a13c10-a53e-42e5-875a-abb1e88ab69c","Type":"ContainerStarted","Data":"34ffae582b16e0bb7c220fa0d77bf3aa7647ca4b6defef072c6e913e3b4a0123"}
Jul 23 17:28:17.082993 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:28:17.082949    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-3f08f93f-kserve-f56c5c8f5-gvb5q" event={"ID":"d7a13c10-a53e-42e5-875a-abb1e88ab69c","Type":"ContainerStarted","Data":"c1fec1f41149c986696d6a266d1a58eeea2ba242f4be4d2a3da9d9cec40246df"}
Jul 23 17:28:17.106329 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:28:17.106248    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-3f08f93f-kserve-f56c5c8f5-gvb5q" podStartSLOduration=1.106231767 podStartE2EDuration="1.106231767s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 17:28:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 17:28:17.104543672 +0000 UTC m=+1392.874829192" watchObservedRunningTime="2026-07-23 17:28:17.106231767 +0000 UTC m=+1392.876517285"
Jul 23 17:28:25.512306 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:28:25.512276    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-3f08f93f-kserve-7fc4c6f4f5-qh8tt"]
Jul 23 17:28:25.516242 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:28:25.516220    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-3f08f93f-kserve-7fc4c6f4f5-qh8tt"
Jul 23 17:28:25.519676 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:28:25.519655    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-3f08f93f-kserve-self-signed-certs\""
Jul 23 17:28:25.533126 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:28:25.533090    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-3f08f93f-kserve-7fc4c6f4f5-qh8tt"]
Jul 23 17:28:25.555717 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:28:25.555691    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:28:25.608684 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:28:25.608657    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wpwmn\" (UniqueName: \"kubernetes.io/projected/1f5d63cc-d7e1-4ae9-8591-3d8bcf291e42-kube-api-access-wpwmn\") pod \"e2e-same-id-b-3f08f93f-kserve-7fc4c6f4f5-qh8tt\" (UID: \"1f5d63cc-d7e1-4ae9-8591-3d8bcf291e42\") " pod="llm/e2e-same-id-b-3f08f93f-kserve-7fc4c6f4f5-qh8tt"
Jul 23 17:28:25.608801 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:28:25.608713    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f5d63cc-d7e1-4ae9-8591-3d8bcf291e42-tmp-dir\") pod \"e2e-same-id-b-3f08f93f-kserve-7fc4c6f4f5-qh8tt\" (UID: \"1f5d63cc-d7e1-4ae9-8591-3d8bcf291e42\") " pod="llm/e2e-same-id-b-3f08f93f-kserve-7fc4c6f4f5-qh8tt"
Jul 23 17:28:25.608801 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:28:25.608738    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f5d63cc-d7e1-4ae9-8591-3d8bcf291e42-tls-certs\") pod \"e2e-same-id-b-3f08f93f-kserve-7fc4c6f4f5-qh8tt\" (UID: \"1f5d63cc-d7e1-4ae9-8591-3d8bcf291e42\") " pod="llm/e2e-same-id-b-3f08f93f-kserve-7fc4c6f4f5-qh8tt"
Jul 23 17:28:25.608897 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:28:25.608838    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1f5d63cc-d7e1-4ae9-8591-3d8bcf291e42-home\") pod \"e2e-same-id-b-3f08f93f-kserve-7fc4c6f4f5-qh8tt\" (UID: \"1f5d63cc-d7e1-4ae9-8591-3d8bcf291e42\") " pod="llm/e2e-same-id-b-3f08f93f-kserve-7fc4c6f4f5-qh8tt"
Jul 23 17:28:25.608897 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:28:25.608861    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1f5d63cc-d7e1-4ae9-8591-3d8bcf291e42-dshm\") pod \"e2e-same-id-b-3f08f93f-kserve-7fc4c6f4f5-qh8tt\" (UID: \"1f5d63cc-d7e1-4ae9-8591-3d8bcf291e42\") " pod="llm/e2e-same-id-b-3f08f93f-kserve-7fc4c6f4f5-qh8tt"
Jul 23 17:28:25.608897 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:28:25.608886    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1f5d63cc-d7e1-4ae9-8591-3d8bcf291e42-model-cache\") pod \"e2e-same-id-b-3f08f93f-kserve-7fc4c6f4f5-qh8tt\" (UID: \"1f5d63cc-d7e1-4ae9-8591-3d8bcf291e42\") " pod="llm/e2e-same-id-b-3f08f93f-kserve-7fc4c6f4f5-qh8tt"
Jul 23 17:28:25.710092 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:28:25.710053    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1f5d63cc-d7e1-4ae9-8591-3d8bcf291e42-home\") pod \"e2e-same-id-b-3f08f93f-kserve-7fc4c6f4f5-qh8tt\" (UID: \"1f5d63cc-d7e1-4ae9-8591-3d8bcf291e42\") " pod="llm/e2e-same-id-b-3f08f93f-kserve-7fc4c6f4f5-qh8tt"
Jul 23 17:28:25.710092 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:28:25.710091    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1f5d63cc-d7e1-4ae9-8591-3d8bcf291e42-dshm\") pod \"e2e-same-id-b-3f08f93f-kserve-7fc4c6f4f5-qh8tt\" (UID: \"1f5d63cc-d7e1-4ae9-8591-3d8bcf291e42\") " pod="llm/e2e-same-id-b-3f08f93f-kserve-7fc4c6f4f5-qh8tt"
Jul 23 17:28:25.710351 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:28:25.710112    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1f5d63cc-d7e1-4ae9-8591-3d8bcf291e42-model-cache\") pod \"e2e-same-id-b-3f08f93f-kserve-7fc4c6f4f5-qh8tt\" (UID: \"1f5d63cc-d7e1-4ae9-8591-3d8bcf291e42\") " pod="llm/e2e-same-id-b-3f08f93f-kserve-7fc4c6f4f5-qh8tt"
Jul 23 17:28:25.710351 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:28:25.710137    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wpwmn\" (UniqueName: \"kubernetes.io/projected/1f5d63cc-d7e1-4ae9-8591-3d8bcf291e42-kube-api-access-wpwmn\") pod \"e2e-same-id-b-3f08f93f-kserve-7fc4c6f4f5-qh8tt\" (UID: \"1f5d63cc-d7e1-4ae9-8591-3d8bcf291e42\") " pod="llm/e2e-same-id-b-3f08f93f-kserve-7fc4c6f4f5-qh8tt"
Jul 23 17:28:25.710351 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:28:25.710189    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f5d63cc-d7e1-4ae9-8591-3d8bcf291e42-tmp-dir\") pod \"e2e-same-id-b-3f08f93f-kserve-7fc4c6f4f5-qh8tt\" (UID: \"1f5d63cc-d7e1-4ae9-8591-3d8bcf291e42\") " pod="llm/e2e-same-id-b-3f08f93f-kserve-7fc4c6f4f5-qh8tt"
Jul 23 17:28:25.710351 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:28:25.710212    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f5d63cc-d7e1-4ae9-8591-3d8bcf291e42-tls-certs\") pod \"e2e-same-id-b-3f08f93f-kserve-7fc4c6f4f5-qh8tt\" (UID: \"1f5d63cc-d7e1-4ae9-8591-3d8bcf291e42\") " pod="llm/e2e-same-id-b-3f08f93f-kserve-7fc4c6f4f5-qh8tt"
Jul 23 17:28:25.710579 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:28:25.710555    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1f5d63cc-d7e1-4ae9-8591-3d8bcf291e42-model-cache\") pod \"e2e-same-id-b-3f08f93f-kserve-7fc4c6f4f5-qh8tt\" (UID: \"1f5d63cc-d7e1-4ae9-8591-3d8bcf291e42\") " pod="llm/e2e-same-id-b-3f08f93f-kserve-7fc4c6f4f5-qh8tt"
Jul 23 17:28:25.710643 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:28:25.710575    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1f5d63cc-d7e1-4ae9-8591-3d8bcf291e42-home\") pod \"e2e-same-id-b-3f08f93f-kserve-7fc4c6f4f5-qh8tt\" (UID: \"1f5d63cc-d7e1-4ae9-8591-3d8bcf291e42\") " pod="llm/e2e-same-id-b-3f08f93f-kserve-7fc4c6f4f5-qh8tt"
Jul 23 17:28:25.710705 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:28:25.710672    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f5d63cc-d7e1-4ae9-8591-3d8bcf291e42-tmp-dir\") pod \"e2e-same-id-b-3f08f93f-kserve-7fc4c6f4f5-qh8tt\" (UID: \"1f5d63cc-d7e1-4ae9-8591-3d8bcf291e42\") " pod="llm/e2e-same-id-b-3f08f93f-kserve-7fc4c6f4f5-qh8tt"
Jul 23 17:28:25.712451 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:28:25.712427    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1f5d63cc-d7e1-4ae9-8591-3d8bcf291e42-dshm\") pod \"e2e-same-id-b-3f08f93f-kserve-7fc4c6f4f5-qh8tt\" (UID: \"1f5d63cc-d7e1-4ae9-8591-3d8bcf291e42\") " pod="llm/e2e-same-id-b-3f08f93f-kserve-7fc4c6f4f5-qh8tt"
Jul 23 17:28:25.712677 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:28:25.712656    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f5d63cc-d7e1-4ae9-8591-3d8bcf291e42-tls-certs\") pod \"e2e-same-id-b-3f08f93f-kserve-7fc4c6f4f5-qh8tt\" (UID: \"1f5d63cc-d7e1-4ae9-8591-3d8bcf291e42\") " pod="llm/e2e-same-id-b-3f08f93f-kserve-7fc4c6f4f5-qh8tt"
Jul 23 17:28:25.718551 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:28:25.718531    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wpwmn\" (UniqueName: \"kubernetes.io/projected/1f5d63cc-d7e1-4ae9-8591-3d8bcf291e42-kube-api-access-wpwmn\") pod \"e2e-same-id-b-3f08f93f-kserve-7fc4c6f4f5-qh8tt\" (UID: \"1f5d63cc-d7e1-4ae9-8591-3d8bcf291e42\") " pod="llm/e2e-same-id-b-3f08f93f-kserve-7fc4c6f4f5-qh8tt"
Jul 23 17:28:25.826840 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:28:25.826768    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-3f08f93f-kserve-7fc4c6f4f5-qh8tt"
Jul 23 17:28:25.969381 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:28:25.969344    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1f5d63cc_d7e1_4ae9_8591_3d8bcf291e42.slice/crio-ea422cfa35384f71c1387f5dfc84f32f0b58129d6ed4b94f8b5599ce98494bed WatchSource:0}: Error finding container ea422cfa35384f71c1387f5dfc84f32f0b58129d6ed4b94f8b5599ce98494bed: Status 404 returned error can't find the container with id ea422cfa35384f71c1387f5dfc84f32f0b58129d6ed4b94f8b5599ce98494bed
Jul 23 17:28:25.971858 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:28:25.971835    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-3f08f93f-kserve-7fc4c6f4f5-qh8tt"]
Jul 23 17:28:26.134432 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:28:26.134340    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-3f08f93f-kserve-7fc4c6f4f5-qh8tt" event={"ID":"1f5d63cc-d7e1-4ae9-8591-3d8bcf291e42","Type":"ContainerStarted","Data":"0e5f4edb087aaa684bf9cd64b1ba4f2e159261f148fa28482551b9d88ba2621a"}
Jul 23 17:28:26.134432 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:28:26.134389    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-3f08f93f-kserve-7fc4c6f4f5-qh8tt" event={"ID":"1f5d63cc-d7e1-4ae9-8591-3d8bcf291e42","Type":"ContainerStarted","Data":"ea422cfa35384f71c1387f5dfc84f32f0b58129d6ed4b94f8b5599ce98494bed"}
Jul 23 17:28:26.825693 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:28:26.825615    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-3f08f93f-kserve-f56c5c8f5-gvb5q"
Jul 23 17:28:26.825693 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:28:26.825661    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-3f08f93f-kserve-f56c5c8f5-gvb5q"
Jul 23 17:28:26.838677 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:28:26.838656    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-3f08f93f-kserve-f56c5c8f5-gvb5q"
Jul 23 17:28:26.874246 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:28:26.874194    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-3f08f93f-kserve-7fc4c6f4f5-qh8tt" podStartSLOduration=1.874169389 podStartE2EDuration="1.874169389s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 17:28:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 17:28:26.171707601 +0000 UTC m=+1401.941993120" watchObservedRunningTime="2026-07-23 17:28:26.874169389 +0000 UTC m=+1402.644454977"
Jul 23 17:28:27.151563 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:28:27.151486    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-3f08f93f-kserve-f56c5c8f5-gvb5q"
Jul 23 17:28:35.827817 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:28:35.827785    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-3f08f93f-kserve-7fc4c6f4f5-qh8tt"
Jul 23 17:28:35.827817 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:28:35.827826    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-3f08f93f-kserve-7fc4c6f4f5-qh8tt"
Jul 23 17:28:35.842554 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:28:35.842526    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-3f08f93f-kserve-7fc4c6f4f5-qh8tt"
Jul 23 17:28:36.199356 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:28:36.199330    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-3f08f93f-kserve-7fc4c6f4f5-qh8tt"
Jul 23 17:28:58.216283 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:28:58.216187    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-3f08f93f-kserve-f56c5c8f5-gvb5q"]
Jul 23 17:28:58.250484 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:28:58.250106    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:28:59.868625 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:28:59.868590    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-3f08f93f-kserve-7fc4c6f4f5-qh8tt"]
Jul 23 17:28:59.892878 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:28:59.892849    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:29:13.240503 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:29:13.240434    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-3f08f93f-kserve-f56c5c8f5-gvb5q" podUID="d7a13c10-a53e-42e5-875a-abb1e88ab69c" containerName="main" containerID="cri-o://34ffae582b16e0bb7c220fa0d77bf3aa7647ca4b6defef072c6e913e3b4a0123" gracePeriod=45
Jul 23 17:29:13.393389 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:29:13.393353    2577 generic.go:356] "Generic (PLEG): container finished" podID="d7a13c10-a53e-42e5-875a-abb1e88ab69c" containerID="34ffae582b16e0bb7c220fa0d77bf3aa7647ca4b6defef072c6e913e3b4a0123" exitCode=0
Jul 23 17:29:13.393560 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:29:13.393432    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-3f08f93f-kserve-f56c5c8f5-gvb5q" event={"ID":"d7a13c10-a53e-42e5-875a-abb1e88ab69c","Type":"ContainerDied","Data":"34ffae582b16e0bb7c220fa0d77bf3aa7647ca4b6defef072c6e913e3b4a0123"}
Jul 23 17:29:13.493737 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:29:13.493672    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-3f08f93f-kserve-f56c5c8f5-gvb5q"
Jul 23 17:29:13.550780 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:29:13.550747    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d7a13c10-a53e-42e5-875a-abb1e88ab69c-tls-certs\") pod \"d7a13c10-a53e-42e5-875a-abb1e88ab69c\" (UID: \"d7a13c10-a53e-42e5-875a-abb1e88ab69c\") "
Jul 23 17:29:13.550780 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:29:13.550781    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d7a13c10-a53e-42e5-875a-abb1e88ab69c-home\") pod \"d7a13c10-a53e-42e5-875a-abb1e88ab69c\" (UID: \"d7a13c10-a53e-42e5-875a-abb1e88ab69c\") "
Jul 23 17:29:13.550986 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:29:13.550808    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d7a13c10-a53e-42e5-875a-abb1e88ab69c-tmp-dir\") pod \"d7a13c10-a53e-42e5-875a-abb1e88ab69c\" (UID: \"d7a13c10-a53e-42e5-875a-abb1e88ab69c\") "
Jul 23 17:29:13.550986 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:29:13.550838    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xzw4z\" (UniqueName: \"kubernetes.io/projected/d7a13c10-a53e-42e5-875a-abb1e88ab69c-kube-api-access-xzw4z\") pod \"d7a13c10-a53e-42e5-875a-abb1e88ab69c\" (UID: \"d7a13c10-a53e-42e5-875a-abb1e88ab69c\") "
Jul 23 17:29:13.550986 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:29:13.550898    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d7a13c10-a53e-42e5-875a-abb1e88ab69c-model-cache\") pod \"d7a13c10-a53e-42e5-875a-abb1e88ab69c\" (UID: \"d7a13c10-a53e-42e5-875a-abb1e88ab69c\") "
Jul 23 17:29:13.550986 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:29:13.550930    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d7a13c10-a53e-42e5-875a-abb1e88ab69c-dshm\") pod \"d7a13c10-a53e-42e5-875a-abb1e88ab69c\" (UID: \"d7a13c10-a53e-42e5-875a-abb1e88ab69c\") "
Jul 23 17:29:13.551211 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:29:13.551054    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7a13c10-a53e-42e5-875a-abb1e88ab69c-home" (OuterVolumeSpecName: "home") pod "d7a13c10-a53e-42e5-875a-abb1e88ab69c" (UID: "d7a13c10-a53e-42e5-875a-abb1e88ab69c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 17:29:13.551211 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:29:13.551106    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7a13c10-a53e-42e5-875a-abb1e88ab69c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d7a13c10-a53e-42e5-875a-abb1e88ab69c" (UID: "d7a13c10-a53e-42e5-875a-abb1e88ab69c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 17:29:13.551211 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:29:13.551158    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7a13c10-a53e-42e5-875a-abb1e88ab69c-model-cache" (OuterVolumeSpecName: "model-cache") pod "d7a13c10-a53e-42e5-875a-abb1e88ab69c" (UID: "d7a13c10-a53e-42e5-875a-abb1e88ab69c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 17:29:13.551390 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:29:13.551235    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d7a13c10-a53e-42e5-875a-abb1e88ab69c-home\") on node \"ip-10-0-129-216.ec2.internal\" DevicePath \"\""
Jul 23 17:29:13.551390 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:29:13.551248    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d7a13c10-a53e-42e5-875a-abb1e88ab69c-tmp-dir\") on node \"ip-10-0-129-216.ec2.internal\" DevicePath \"\""
Jul 23 17:29:13.551390 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:29:13.551281    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d7a13c10-a53e-42e5-875a-abb1e88ab69c-model-cache\") on node \"ip-10-0-129-216.ec2.internal\" DevicePath \"\""
Jul 23 17:29:13.553043 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:29:13.553011    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7a13c10-a53e-42e5-875a-abb1e88ab69c-dshm" (OuterVolumeSpecName: "dshm") pod "d7a13c10-a53e-42e5-875a-abb1e88ab69c" (UID: "d7a13c10-a53e-42e5-875a-abb1e88ab69c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 17:29:13.553160 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:29:13.553066    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d7a13c10-a53e-42e5-875a-abb1e88ab69c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d7a13c10-a53e-42e5-875a-abb1e88ab69c" (UID: "d7a13c10-a53e-42e5-875a-abb1e88ab69c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 17:29:13.553160 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:29:13.553100    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7a13c10-a53e-42e5-875a-abb1e88ab69c-kube-api-access-xzw4z" (OuterVolumeSpecName: "kube-api-access-xzw4z") pod "d7a13c10-a53e-42e5-875a-abb1e88ab69c" (UID: "d7a13c10-a53e-42e5-875a-abb1e88ab69c"). InnerVolumeSpecName "kube-api-access-xzw4z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 17:29:13.652357 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:29:13.652324    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d7a13c10-a53e-42e5-875a-abb1e88ab69c-dshm\") on node \"ip-10-0-129-216.ec2.internal\" DevicePath \"\""
Jul 23 17:29:13.652357 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:29:13.652353    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d7a13c10-a53e-42e5-875a-abb1e88ab69c-tls-certs\") on node \"ip-10-0-129-216.ec2.internal\" DevicePath \"\""
Jul 23 17:29:13.652357 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:29:13.652364    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xzw4z\" (UniqueName: \"kubernetes.io/projected/d7a13c10-a53e-42e5-875a-abb1e88ab69c-kube-api-access-xzw4z\") on node \"ip-10-0-129-216.ec2.internal\" DevicePath \"\""
Jul 23 17:29:14.406859 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:29:14.406827    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-3f08f93f-kserve-f56c5c8f5-gvb5q"
Jul 23 17:29:14.406859 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:29:14.406853    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-3f08f93f-kserve-f56c5c8f5-gvb5q" event={"ID":"d7a13c10-a53e-42e5-875a-abb1e88ab69c","Type":"ContainerDied","Data":"c1fec1f41149c986696d6a266d1a58eeea2ba242f4be4d2a3da9d9cec40246df"}
Jul 23 17:29:14.407345 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:29:14.406899    2577 scope.go:117] "RemoveContainer" containerID="34ffae582b16e0bb7c220fa0d77bf3aa7647ca4b6defef072c6e913e3b4a0123"
Jul 23 17:29:14.429139 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:29:14.429115    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-3f08f93f-kserve-f56c5c8f5-gvb5q"]
Jul 23 17:29:14.432179 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:29:14.432155    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-3f08f93f-kserve-f56c5c8f5-gvb5q"]
Jul 23 17:29:14.766053 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:29:14.766019    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d7a13c10-a53e-42e5-875a-abb1e88ab69c" path="/var/lib/kubelet/pods/d7a13c10-a53e-42e5-875a-abb1e88ab69c/volumes"
Jul 23 17:29:14.888613 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:29:14.888577    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-3f08f93f-kserve-7fc4c6f4f5-qh8tt" podUID="1f5d63cc-d7e1-4ae9-8591-3d8bcf291e42" containerName="main" containerID="cri-o://0e5f4edb087aaa684bf9cd64b1ba4f2e159261f148fa28482551b9d88ba2621a" gracePeriod=45
Jul 23 17:29:15.158850 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:29:15.158819    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-3f08f93f-kserve-7fc4c6f4f5-qh8tt"
Jul 23 17:29:15.267814 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:29:15.267772    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1f5d63cc-d7e1-4ae9-8591-3d8bcf291e42-model-cache\") pod \"1f5d63cc-d7e1-4ae9-8591-3d8bcf291e42\" (UID: \"1f5d63cc-d7e1-4ae9-8591-3d8bcf291e42\") "
Jul 23 17:29:15.267814 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:29:15.267829    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1f5d63cc-d7e1-4ae9-8591-3d8bcf291e42-dshm\") pod \"1f5d63cc-d7e1-4ae9-8591-3d8bcf291e42\" (UID: \"1f5d63cc-d7e1-4ae9-8591-3d8bcf291e42\") "
Jul 23 17:29:15.268018 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:29:15.267849    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1f5d63cc-d7e1-4ae9-8591-3d8bcf291e42-home\") pod \"1f5d63cc-d7e1-4ae9-8591-3d8bcf291e42\" (UID: \"1f5d63cc-d7e1-4ae9-8591-3d8bcf291e42\") "
Jul 23 17:29:15.268018 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:29:15.267871    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f5d63cc-d7e1-4ae9-8591-3d8bcf291e42-tls-certs\") pod \"1f5d63cc-d7e1-4ae9-8591-3d8bcf291e42\" (UID: \"1f5d63cc-d7e1-4ae9-8591-3d8bcf291e42\") "
Jul 23 17:29:15.268018 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:29:15.267952    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wpwmn\" (UniqueName: \"kubernetes.io/projected/1f5d63cc-d7e1-4ae9-8591-3d8bcf291e42-kube-api-access-wpwmn\") pod \"1f5d63cc-d7e1-4ae9-8591-3d8bcf291e42\" (UID: \"1f5d63cc-d7e1-4ae9-8591-3d8bcf291e42\") "
Jul 23 17:29:15.268018 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:29:15.267972    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f5d63cc-d7e1-4ae9-8591-3d8bcf291e42-tmp-dir\") pod \"1f5d63cc-d7e1-4ae9-8591-3d8bcf291e42\" (UID: \"1f5d63cc-d7e1-4ae9-8591-3d8bcf291e42\") "
Jul 23 17:29:15.268198 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:29:15.268104    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f5d63cc-d7e1-4ae9-8591-3d8bcf291e42-model-cache" (OuterVolumeSpecName: "model-cache") pod "1f5d63cc-d7e1-4ae9-8591-3d8bcf291e42" (UID: "1f5d63cc-d7e1-4ae9-8591-3d8bcf291e42"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 17:29:15.268198 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:29:15.268120    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f5d63cc-d7e1-4ae9-8591-3d8bcf291e42-home" (OuterVolumeSpecName: "home") pod "1f5d63cc-d7e1-4ae9-8591-3d8bcf291e42" (UID: "1f5d63cc-d7e1-4ae9-8591-3d8bcf291e42"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 17:29:15.268351 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:29:15.268234    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1f5d63cc-d7e1-4ae9-8591-3d8bcf291e42-model-cache\") on node \"ip-10-0-129-216.ec2.internal\" DevicePath \"\""
Jul 23 17:29:15.268351 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:29:15.268247    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1f5d63cc-d7e1-4ae9-8591-3d8bcf291e42-home\") on node \"ip-10-0-129-216.ec2.internal\" DevicePath \"\""
Jul 23 17:29:15.268439 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:29:15.268359    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f5d63cc-d7e1-4ae9-8591-3d8bcf291e42-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1f5d63cc-d7e1-4ae9-8591-3d8bcf291e42" (UID: "1f5d63cc-d7e1-4ae9-8591-3d8bcf291e42"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 17:29:15.269977 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:29:15.269956    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f5d63cc-d7e1-4ae9-8591-3d8bcf291e42-dshm" (OuterVolumeSpecName: "dshm") pod "1f5d63cc-d7e1-4ae9-8591-3d8bcf291e42" (UID: "1f5d63cc-d7e1-4ae9-8591-3d8bcf291e42"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 17:29:15.269977 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:29:15.269963    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f5d63cc-d7e1-4ae9-8591-3d8bcf291e42-kube-api-access-wpwmn" (OuterVolumeSpecName: "kube-api-access-wpwmn") pod "1f5d63cc-d7e1-4ae9-8591-3d8bcf291e42" (UID: "1f5d63cc-d7e1-4ae9-8591-3d8bcf291e42"). InnerVolumeSpecName "kube-api-access-wpwmn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 17:29:15.270136 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:29:15.269995    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f5d63cc-d7e1-4ae9-8591-3d8bcf291e42-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1f5d63cc-d7e1-4ae9-8591-3d8bcf291e42" (UID: "1f5d63cc-d7e1-4ae9-8591-3d8bcf291e42"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 17:29:15.369589 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:29:15.369509    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wpwmn\" (UniqueName: \"kubernetes.io/projected/1f5d63cc-d7e1-4ae9-8591-3d8bcf291e42-kube-api-access-wpwmn\") on node \"ip-10-0-129-216.ec2.internal\" DevicePath \"\""
Jul 23 17:29:15.369589 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:29:15.369548    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f5d63cc-d7e1-4ae9-8591-3d8bcf291e42-tmp-dir\") on node \"ip-10-0-129-216.ec2.internal\" DevicePath \"\""
Jul 23 17:29:15.369589 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:29:15.369558    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1f5d63cc-d7e1-4ae9-8591-3d8bcf291e42-dshm\") on node \"ip-10-0-129-216.ec2.internal\" DevicePath \"\""
Jul 23 17:29:15.369589 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:29:15.369567    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f5d63cc-d7e1-4ae9-8591-3d8bcf291e42-tls-certs\") on node \"ip-10-0-129-216.ec2.internal\" DevicePath \"\""
Jul 23 17:29:15.413837 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:29:15.413806    2577 generic.go:356] "Generic (PLEG): container finished" podID="1f5d63cc-d7e1-4ae9-8591-3d8bcf291e42" containerID="0e5f4edb087aaa684bf9cd64b1ba4f2e159261f148fa28482551b9d88ba2621a" exitCode=0
Jul 23 17:29:15.414224 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:29:15.413847    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-3f08f93f-kserve-7fc4c6f4f5-qh8tt"
Jul 23 17:29:15.414224 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:29:15.413885    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-3f08f93f-kserve-7fc4c6f4f5-qh8tt" event={"ID":"1f5d63cc-d7e1-4ae9-8591-3d8bcf291e42","Type":"ContainerDied","Data":"0e5f4edb087aaa684bf9cd64b1ba4f2e159261f148fa28482551b9d88ba2621a"}
Jul 23 17:29:15.414224 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:29:15.413926    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-3f08f93f-kserve-7fc4c6f4f5-qh8tt" event={"ID":"1f5d63cc-d7e1-4ae9-8591-3d8bcf291e42","Type":"ContainerDied","Data":"ea422cfa35384f71c1387f5dfc84f32f0b58129d6ed4b94f8b5599ce98494bed"}
Jul 23 17:29:15.414224 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:29:15.413951    2577 scope.go:117] "RemoveContainer" containerID="0e5f4edb087aaa684bf9cd64b1ba4f2e159261f148fa28482551b9d88ba2621a"
Jul 23 17:29:15.423043 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:29:15.423022    2577 scope.go:117] "RemoveContainer" containerID="0e5f4edb087aaa684bf9cd64b1ba4f2e159261f148fa28482551b9d88ba2621a"
Jul 23 17:29:15.423298 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:29:15.423271    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e5f4edb087aaa684bf9cd64b1ba4f2e159261f148fa28482551b9d88ba2621a\": container with ID starting with 0e5f4edb087aaa684bf9cd64b1ba4f2e159261f148fa28482551b9d88ba2621a not found: ID does not exist" containerID="0e5f4edb087aaa684bf9cd64b1ba4f2e159261f148fa28482551b9d88ba2621a"
Jul 23 17:29:15.423402 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:29:15.423304    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e5f4edb087aaa684bf9cd64b1ba4f2e159261f148fa28482551b9d88ba2621a"} err="failed to get container status \"0e5f4edb087aaa684bf9cd64b1ba4f2e159261f148fa28482551b9d88ba2621a\": rpc error: code = NotFound desc = could not find container \"0e5f4edb087aaa684bf9cd64b1ba4f2e159261f148fa28482551b9d88ba2621a\": container with ID starting with 0e5f4edb087aaa684bf9cd64b1ba4f2e159261f148fa28482551b9d88ba2621a not found: ID does not exist"
Jul 23 17:29:15.437357 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:29:15.437334    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-3f08f93f-kserve-7fc4c6f4f5-qh8tt"]
Jul 23 17:29:15.441635 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:29:15.441613    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-3f08f93f-kserve-7fc4c6f4f5-qh8tt"]
Jul 23 17:29:16.766096 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:29:16.766063    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1f5d63cc-d7e1-4ae9-8591-3d8bcf291e42" path="/var/lib/kubelet/pods/1f5d63cc-d7e1-4ae9-8591-3d8bcf291e42/volumes"
Jul 23 17:31:40.364866 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:31:40.364831    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:32:11.382982 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:32:11.382893    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:32:35.119678 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:32:35.119640    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:32:38.430772 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:32:38.430722    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:32:38.555153 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:32:38.555121    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:32:38.941891 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:32:38.941858    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:32:45.990508 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:32:45.990441    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:32:46.157894 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:32:46.157862    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:32:46.338957 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:32:46.338875    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:32:46.744315 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:32:46.744285    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:32:47.189291 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:32:47.189191    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:32:48.826925 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:32:48.826861    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:32:48.977175 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:32:48.977135    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:32:48.996971 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:32:48.996942    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-dbb3204a-7f8bc8d7d8-jjpmf"]
Jul 23 17:32:48.997508 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:32:48.997487    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d7a13c10-a53e-42e5-875a-abb1e88ab69c" containerName="main"
Jul 23 17:32:48.997508 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:32:48.997506    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="d7a13c10-a53e-42e5-875a-abb1e88ab69c" containerName="main"
Jul 23 17:32:48.997656 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:32:48.997540    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1f5d63cc-d7e1-4ae9-8591-3d8bcf291e42" containerName="main"
Jul 23 17:32:48.997656 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:32:48.997546    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="1f5d63cc-d7e1-4ae9-8591-3d8bcf291e42" containerName="main"
Jul 23 17:32:48.997656 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:32:48.997616    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="1f5d63cc-d7e1-4ae9-8591-3d8bcf291e42" containerName="main"
Jul 23 17:32:48.997656 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:32:48.997628    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="d7a13c10-a53e-42e5-875a-abb1e88ab69c" containerName="main"
Jul 23 17:32:49.000708 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:32:49.000693    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-dbb3204a-7f8bc8d7d8-jjpmf"
Jul 23 17:32:49.006721 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:32:49.006697    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-dbb3204a-serving-cert\""
Jul 23 17:32:49.021496 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:32:49.021469    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-dbb3204a-7f8bc8d7d8-jjpmf"]
Jul 23 17:32:49.147639 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:32:49.147549    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2025d5a8-a60d-4635-9adf-c9bbd007141e-openshift-service-ca\") pod \"maas-api-e2e-ait-dbb3204a-7f8bc8d7d8-jjpmf\" (UID: \"2025d5a8-a60d-4635-9adf-c9bbd007141e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-dbb3204a-7f8bc8d7d8-jjpmf"
Jul 23 17:32:49.147802 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:32:49.147658    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2025d5a8-a60d-4635-9adf-c9bbd007141e-maas-api-tls\") pod \"maas-api-e2e-ait-dbb3204a-7f8bc8d7d8-jjpmf\" (UID: \"2025d5a8-a60d-4635-9adf-c9bbd007141e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-dbb3204a-7f8bc8d7d8-jjpmf"
Jul 23 17:32:49.147802 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:32:49.147687    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2zg2\" (UniqueName: \"kubernetes.io/projected/2025d5a8-a60d-4635-9adf-c9bbd007141e-kube-api-access-h2zg2\") pod \"maas-api-e2e-ait-dbb3204a-7f8bc8d7d8-jjpmf\" (UID: \"2025d5a8-a60d-4635-9adf-c9bbd007141e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-dbb3204a-7f8bc8d7d8-jjpmf"
Jul 23 17:32:49.194645 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:32:49.194598    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:32:49.248547 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:32:49.248523    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2025d5a8-a60d-4635-9adf-c9bbd007141e-maas-api-tls\") pod \"maas-api-e2e-ait-dbb3204a-7f8bc8d7d8-jjpmf\" (UID: \"2025d5a8-a60d-4635-9adf-c9bbd007141e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-dbb3204a-7f8bc8d7d8-jjpmf"
Jul 23 17:32:49.248718 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:32:49.248562    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2zg2\" (UniqueName: \"kubernetes.io/projected/2025d5a8-a60d-4635-9adf-c9bbd007141e-kube-api-access-h2zg2\") pod \"maas-api-e2e-ait-dbb3204a-7f8bc8d7d8-jjpmf\" (UID: \"2025d5a8-a60d-4635-9adf-c9bbd007141e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-dbb3204a-7f8bc8d7d8-jjpmf"
Jul 23 17:32:49.248718 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:32:49.248582    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2025d5a8-a60d-4635-9adf-c9bbd007141e-openshift-service-ca\") pod \"maas-api-e2e-ait-dbb3204a-7f8bc8d7d8-jjpmf\" (UID: \"2025d5a8-a60d-4635-9adf-c9bbd007141e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-dbb3204a-7f8bc8d7d8-jjpmf"
Jul 23 17:32:49.248718 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:32:49.248677    2577 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-dbb3204a-serving-cert: secret "maas-api-e2e-ait-dbb3204a-serving-cert" not found
Jul 23 17:32:49.248887 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:32:49.248744    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2025d5a8-a60d-4635-9adf-c9bbd007141e-maas-api-tls podName:2025d5a8-a60d-4635-9adf-c9bbd007141e nodeName:}" failed. No retries permitted until 2026-07-23 17:32:49.748726196 +0000 UTC m=+1665.519011695 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/2025d5a8-a60d-4635-9adf-c9bbd007141e-maas-api-tls") pod "maas-api-e2e-ait-dbb3204a-7f8bc8d7d8-jjpmf" (UID: "2025d5a8-a60d-4635-9adf-c9bbd007141e") : secret "maas-api-e2e-ait-dbb3204a-serving-cert" not found
Jul 23 17:32:49.249186 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:32:49.249168    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2025d5a8-a60d-4635-9adf-c9bbd007141e-openshift-service-ca\") pod \"maas-api-e2e-ait-dbb3204a-7f8bc8d7d8-jjpmf\" (UID: \"2025d5a8-a60d-4635-9adf-c9bbd007141e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-dbb3204a-7f8bc8d7d8-jjpmf"
Jul 23 17:32:49.271905 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:32:49.271876    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2zg2\" (UniqueName: \"kubernetes.io/projected/2025d5a8-a60d-4635-9adf-c9bbd007141e-kube-api-access-h2zg2\") pod \"maas-api-e2e-ait-dbb3204a-7f8bc8d7d8-jjpmf\" (UID: \"2025d5a8-a60d-4635-9adf-c9bbd007141e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-dbb3204a-7f8bc8d7d8-jjpmf"
Jul 23 17:32:49.490238 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:32:49.490192    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:32:49.755016 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:32:49.754932    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2025d5a8-a60d-4635-9adf-c9bbd007141e-maas-api-tls\") pod \"maas-api-e2e-ait-dbb3204a-7f8bc8d7d8-jjpmf\" (UID: \"2025d5a8-a60d-4635-9adf-c9bbd007141e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-dbb3204a-7f8bc8d7d8-jjpmf"
Jul 23 17:32:49.757222 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:32:49.757193    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2025d5a8-a60d-4635-9adf-c9bbd007141e-maas-api-tls\") pod \"maas-api-e2e-ait-dbb3204a-7f8bc8d7d8-jjpmf\" (UID: \"2025d5a8-a60d-4635-9adf-c9bbd007141e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-dbb3204a-7f8bc8d7d8-jjpmf"
Jul 23 17:32:49.912089 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:32:49.912050    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-dbb3204a-7f8bc8d7d8-jjpmf"
Jul 23 17:32:50.079772 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:32:50.079741    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-dbb3204a-7f8bc8d7d8-jjpmf"]
Jul 23 17:32:50.080372 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:32:50.080348    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2025d5a8_a60d_4635_9adf_c9bbd007141e.slice/crio-87857be26d5c6d206102c625745f309116eb7befae2fb26b49c578650e87b0cc WatchSource:0}: Error finding container 87857be26d5c6d206102c625745f309116eb7befae2fb26b49c578650e87b0cc: Status 404 returned error can't find the container with id 87857be26d5c6d206102c625745f309116eb7befae2fb26b49c578650e87b0cc
Jul 23 17:32:50.081778 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:32:50.081759    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 23 17:32:50.214393 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:32:50.214357    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:32:50.355607 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:32:50.355529    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:32:50.587555 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:32:50.587522    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-dbb3204a-7f8bc8d7d8-jjpmf" event={"ID":"2025d5a8-a60d-4635-9adf-c9bbd007141e","Type":"ContainerStarted","Data":"87857be26d5c6d206102c625745f309116eb7befae2fb26b49c578650e87b0cc"}
Jul 23 17:32:51.594884 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:32:51.594846    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-dbb3204a-7f8bc8d7d8-jjpmf" event={"ID":"2025d5a8-a60d-4635-9adf-c9bbd007141e","Type":"ContainerStarted","Data":"442a055423d6a45f8031793ea31bc6813644292a04812334b7e96b8a0fbe8a65"}
Jul 23 17:32:51.595354 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:32:51.594903    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-dbb3204a-7f8bc8d7d8-jjpmf"
Jul 23 17:32:51.634341 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:32:51.634286    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-dbb3204a-7f8bc8d7d8-jjpmf" podStartSLOduration=2.953962464 podStartE2EDuration="3.634244123s" totalImagesPullingTime="680.281659ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 17:32:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 17:32:51.632878555 +0000 UTC m=+1667.403164074" watchObservedRunningTime="2026-07-23 17:32:51.634244123 +0000 UTC m=+1667.404529642"
Jul 23 17:32:57.605832 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:32:57.605801    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-dbb3204a-7f8bc8d7d8-jjpmf"
Jul 23 17:33:24.997468 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:33:24.997433    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-dbb3204a-7f8bc8d7d8-jjpmf"]
Jul 23 17:33:24.998122 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:33:24.997647    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-dbb3204a-7f8bc8d7d8-jjpmf" podUID="2025d5a8-a60d-4635-9adf-c9bbd007141e" containerName="maas-api" containerID="cri-o://442a055423d6a45f8031793ea31bc6813644292a04812334b7e96b8a0fbe8a65" gracePeriod=30
Jul 23 17:33:25.030655 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:33:25.030608    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:33:25.249313 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:33:25.249210    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-dbb3204a-7f8bc8d7d8-jjpmf"
Jul 23 17:33:25.358936 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:33:25.358904    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2025d5a8-a60d-4635-9adf-c9bbd007141e-openshift-service-ca\") pod \"2025d5a8-a60d-4635-9adf-c9bbd007141e\" (UID: \"2025d5a8-a60d-4635-9adf-c9bbd007141e\") "
Jul 23 17:33:25.359114 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:33:25.358968    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2025d5a8-a60d-4635-9adf-c9bbd007141e-maas-api-tls\") pod \"2025d5a8-a60d-4635-9adf-c9bbd007141e\" (UID: \"2025d5a8-a60d-4635-9adf-c9bbd007141e\") "
Jul 23 17:33:25.359114 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:33:25.359009    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h2zg2\" (UniqueName: \"kubernetes.io/projected/2025d5a8-a60d-4635-9adf-c9bbd007141e-kube-api-access-h2zg2\") pod \"2025d5a8-a60d-4635-9adf-c9bbd007141e\" (UID: \"2025d5a8-a60d-4635-9adf-c9bbd007141e\") "
Jul 23 17:33:25.359316 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:33:25.359292    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2025d5a8-a60d-4635-9adf-c9bbd007141e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "2025d5a8-a60d-4635-9adf-c9bbd007141e" (UID: "2025d5a8-a60d-4635-9adf-c9bbd007141e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 17:33:25.361209 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:33:25.361179    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2025d5a8-a60d-4635-9adf-c9bbd007141e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "2025d5a8-a60d-4635-9adf-c9bbd007141e" (UID: "2025d5a8-a60d-4635-9adf-c9bbd007141e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 17:33:25.361332 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:33:25.361274    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2025d5a8-a60d-4635-9adf-c9bbd007141e-kube-api-access-h2zg2" (OuterVolumeSpecName: "kube-api-access-h2zg2") pod "2025d5a8-a60d-4635-9adf-c9bbd007141e" (UID: "2025d5a8-a60d-4635-9adf-c9bbd007141e"). InnerVolumeSpecName "kube-api-access-h2zg2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 17:33:25.460236 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:33:25.460200    2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2025d5a8-a60d-4635-9adf-c9bbd007141e-openshift-service-ca\") on node \"ip-10-0-129-216.ec2.internal\" DevicePath \"\""
Jul 23 17:33:25.460236 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:33:25.460231    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2025d5a8-a60d-4635-9adf-c9bbd007141e-maas-api-tls\") on node \"ip-10-0-129-216.ec2.internal\" DevicePath \"\""
Jul 23 17:33:25.460236 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:33:25.460245    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h2zg2\" (UniqueName: \"kubernetes.io/projected/2025d5a8-a60d-4635-9adf-c9bbd007141e-kube-api-access-h2zg2\") on node \"ip-10-0-129-216.ec2.internal\" DevicePath \"\""
Jul 23 17:33:25.783472 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:33:25.783440    2577 generic.go:356] "Generic (PLEG): container finished" podID="2025d5a8-a60d-4635-9adf-c9bbd007141e" containerID="442a055423d6a45f8031793ea31bc6813644292a04812334b7e96b8a0fbe8a65" exitCode=0
Jul 23 17:33:25.783645 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:33:25.783487    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-dbb3204a-7f8bc8d7d8-jjpmf"
Jul 23 17:33:25.783645 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:33:25.783528    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-dbb3204a-7f8bc8d7d8-jjpmf" event={"ID":"2025d5a8-a60d-4635-9adf-c9bbd007141e","Type":"ContainerDied","Data":"442a055423d6a45f8031793ea31bc6813644292a04812334b7e96b8a0fbe8a65"}
Jul 23 17:33:25.783645 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:33:25.783582    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-dbb3204a-7f8bc8d7d8-jjpmf" event={"ID":"2025d5a8-a60d-4635-9adf-c9bbd007141e","Type":"ContainerDied","Data":"87857be26d5c6d206102c625745f309116eb7befae2fb26b49c578650e87b0cc"}
Jul 23 17:33:25.783645 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:33:25.783609    2577 scope.go:117] "RemoveContainer" containerID="442a055423d6a45f8031793ea31bc6813644292a04812334b7e96b8a0fbe8a65"
Jul 23 17:33:25.798957 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:33:25.798920    2577 scope.go:117] "RemoveContainer" containerID="442a055423d6a45f8031793ea31bc6813644292a04812334b7e96b8a0fbe8a65"
Jul 23 17:33:25.799220 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:33:25.799198    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"442a055423d6a45f8031793ea31bc6813644292a04812334b7e96b8a0fbe8a65\": container with ID starting with 442a055423d6a45f8031793ea31bc6813644292a04812334b7e96b8a0fbe8a65 not found: ID does not exist" containerID="442a055423d6a45f8031793ea31bc6813644292a04812334b7e96b8a0fbe8a65"
Jul 23 17:33:25.799313 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:33:25.799232    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"442a055423d6a45f8031793ea31bc6813644292a04812334b7e96b8a0fbe8a65"} err="failed to get container status \"442a055423d6a45f8031793ea31bc6813644292a04812334b7e96b8a0fbe8a65\": rpc error: code = NotFound desc = could not find container \"442a055423d6a45f8031793ea31bc6813644292a04812334b7e96b8a0fbe8a65\": container with ID starting with 442a055423d6a45f8031793ea31bc6813644292a04812334b7e96b8a0fbe8a65 not found: ID does not exist"
Jul 23 17:33:25.810114 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:33:25.810093    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-dbb3204a-7f8bc8d7d8-jjpmf"]
Jul 23 17:33:25.812851 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:33:25.812828    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-dbb3204a-7f8bc8d7d8-jjpmf"]
Jul 23 17:33:26.766505 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:33:26.766423    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2025d5a8-a60d-4635-9adf-c9bbd007141e" path="/var/lib/kubelet/pods/2025d5a8-a60d-4635-9adf-c9bbd007141e/volumes"
Jul 23 17:33:29.587049 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:33:29.587011    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:33:29.765757 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:33:29.765724    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:33:30.086399 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:33:30.086362    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:33:30.315916 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:33:30.315884    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:33:30.404362 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:33:30.404287    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:33:30.539959 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:33:30.539928    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:33:30.883657 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:33:30.883603    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:33:31.399329 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:33:31.399298    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:33:33.147771 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:33:33.147728    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:33:33.308522 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:33:33.308492    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-ab1734b0-59c5888b6f-zvgbd"]
Jul 23 17:33:33.308952 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:33:33.308936    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2025d5a8-a60d-4635-9adf-c9bbd007141e" containerName="maas-api"
Jul 23 17:33:33.308997 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:33:33.308956    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="2025d5a8-a60d-4635-9adf-c9bbd007141e" containerName="maas-api"
Jul 23 17:33:33.309052 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:33:33.309042    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="2025d5a8-a60d-4635-9adf-c9bbd007141e" containerName="maas-api"
Jul 23 17:33:33.313497 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:33:33.313475    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ab1734b0-59c5888b6f-zvgbd"
Jul 23 17:33:33.315930 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:33:33.315908    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-ab1734b0-serving-cert\""
Jul 23 17:33:33.322998 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:33:33.322973    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-ab1734b0-59c5888b6f-zvgbd"]
Jul 23 17:33:33.344765 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:33:33.344733    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:33:33.436568 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:33:33.436481    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cd76e99a-e2af-46fb-9165-63d019330bc5-maas-api-tls\") pod \"maas-api-e2e-migrate-ab1734b0-59c5888b6f-zvgbd\" (UID: \"cd76e99a-e2af-46fb-9165-63d019330bc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ab1734b0-59c5888b6f-zvgbd"
Jul 23 17:33:33.436727 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:33:33.436592    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd76e99a-e2af-46fb-9165-63d019330bc5-openshift-service-ca\") pod \"maas-api-e2e-migrate-ab1734b0-59c5888b6f-zvgbd\" (UID: \"cd76e99a-e2af-46fb-9165-63d019330bc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ab1734b0-59c5888b6f-zvgbd"
Jul 23 17:33:33.436727 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:33:33.436672    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2lxz\" (UniqueName: \"kubernetes.io/projected/cd76e99a-e2af-46fb-9165-63d019330bc5-kube-api-access-p2lxz\") pod \"maas-api-e2e-migrate-ab1734b0-59c5888b6f-zvgbd\" (UID: \"cd76e99a-e2af-46fb-9165-63d019330bc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ab1734b0-59c5888b6f-zvgbd"
Jul 23 17:33:33.537314 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:33:33.537274    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd76e99a-e2af-46fb-9165-63d019330bc5-openshift-service-ca\") pod \"maas-api-e2e-migrate-ab1734b0-59c5888b6f-zvgbd\" (UID: \"cd76e99a-e2af-46fb-9165-63d019330bc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ab1734b0-59c5888b6f-zvgbd"
Jul 23 17:33:33.537495 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:33:33.537347    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2lxz\" (UniqueName: \"kubernetes.io/projected/cd76e99a-e2af-46fb-9165-63d019330bc5-kube-api-access-p2lxz\") pod \"maas-api-e2e-migrate-ab1734b0-59c5888b6f-zvgbd\" (UID: \"cd76e99a-e2af-46fb-9165-63d019330bc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ab1734b0-59c5888b6f-zvgbd"
Jul 23 17:33:33.537495 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:33:33.537387    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cd76e99a-e2af-46fb-9165-63d019330bc5-maas-api-tls\") pod \"maas-api-e2e-migrate-ab1734b0-59c5888b6f-zvgbd\" (UID: \"cd76e99a-e2af-46fb-9165-63d019330bc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ab1734b0-59c5888b6f-zvgbd"
Jul 23 17:33:33.537629 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:33:33.537499    2577 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-ab1734b0-serving-cert: secret "maas-api-e2e-migrate-ab1734b0-serving-cert" not found
Jul 23 17:33:33.537629 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:33:33.537560    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd76e99a-e2af-46fb-9165-63d019330bc5-maas-api-tls podName:cd76e99a-e2af-46fb-9165-63d019330bc5 nodeName:}" failed. No retries permitted until 2026-07-23 17:33:34.03754169 +0000 UTC m=+1709.807827187 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/cd76e99a-e2af-46fb-9165-63d019330bc5-maas-api-tls") pod "maas-api-e2e-migrate-ab1734b0-59c5888b6f-zvgbd" (UID: "cd76e99a-e2af-46fb-9165-63d019330bc5") : secret "maas-api-e2e-migrate-ab1734b0-serving-cert" not found
Jul 23 17:33:33.537973 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:33:33.537950    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd76e99a-e2af-46fb-9165-63d019330bc5-openshift-service-ca\") pod \"maas-api-e2e-migrate-ab1734b0-59c5888b6f-zvgbd\" (UID: \"cd76e99a-e2af-46fb-9165-63d019330bc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ab1734b0-59c5888b6f-zvgbd"
Jul 23 17:33:33.557384 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:33:33.557361    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2lxz\" (UniqueName: \"kubernetes.io/projected/cd76e99a-e2af-46fb-9165-63d019330bc5-kube-api-access-p2lxz\") pod \"maas-api-e2e-migrate-ab1734b0-59c5888b6f-zvgbd\" (UID: \"cd76e99a-e2af-46fb-9165-63d019330bc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ab1734b0-59c5888b6f-zvgbd"
Jul 23 17:33:33.824992 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:33:33.824955    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:33:33.922659 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:33:33.922631    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:33:34.042475 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:33:34.042445    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cd76e99a-e2af-46fb-9165-63d019330bc5-maas-api-tls\") pod \"maas-api-e2e-migrate-ab1734b0-59c5888b6f-zvgbd\" (UID: \"cd76e99a-e2af-46fb-9165-63d019330bc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ab1734b0-59c5888b6f-zvgbd"
Jul 23 17:33:34.045051 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:33:34.045026    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cd76e99a-e2af-46fb-9165-63d019330bc5-maas-api-tls\") pod \"maas-api-e2e-migrate-ab1734b0-59c5888b6f-zvgbd\" (UID: \"cd76e99a-e2af-46fb-9165-63d019330bc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ab1734b0-59c5888b6f-zvgbd"
Jul 23 17:33:34.226822 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:33:34.226777    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ab1734b0-59c5888b6f-zvgbd"
Jul 23 17:33:34.367497 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:33:34.367466    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-ab1734b0-59c5888b6f-zvgbd"]
Jul 23 17:33:34.367947 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:33:34.367918    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd76e99a_e2af_46fb_9165_63d019330bc5.slice/crio-1153666ba853fd43179578008798436675c56695855ab8cbc1fc0e2a63555db3 WatchSource:0}: Error finding container 1153666ba853fd43179578008798436675c56695855ab8cbc1fc0e2a63555db3: Status 404 returned error can't find the container with id 1153666ba853fd43179578008798436675c56695855ab8cbc1fc0e2a63555db3
Jul 23 17:33:34.834371 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:33:34.834322    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ab1734b0-59c5888b6f-zvgbd" event={"ID":"cd76e99a-e2af-46fb-9165-63d019330bc5","Type":"ContainerStarted","Data":"1153666ba853fd43179578008798436675c56695855ab8cbc1fc0e2a63555db3"}
Jul 23 17:33:35.842169 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:33:35.842134    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ab1734b0-59c5888b6f-zvgbd" event={"ID":"cd76e99a-e2af-46fb-9165-63d019330bc5","Type":"ContainerStarted","Data":"ec55f00d537201573476bd9d9f9593aee3840915c1c78f298386c8dcc56e820b"}
Jul 23 17:33:35.842581 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:33:35.842235    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ab1734b0-59c5888b6f-zvgbd"
Jul 23 17:33:35.860060 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:33:35.860011    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ab1734b0-59c5888b6f-zvgbd" podStartSLOduration=2.372103277 podStartE2EDuration="2.859996197s" totalImagesPullingTime="487.89292ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 17:33:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 17:33:35.859142426 +0000 UTC m=+1711.629427946" watchObservedRunningTime="2026-07-23 17:33:35.859996197 +0000 UTC m=+1711.630281717"
Jul 23 17:33:41.853351 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:33:41.853320    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ab1734b0-59c5888b6f-zvgbd"
Jul 23 17:33:58.176549 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:33:58.176511    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ab1734b0-svdrx"]
Jul 23 17:33:58.180325 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:33:58.180305    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ab1734b0-svdrx"
Jul 23 17:33:58.183070 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:33:58.183052    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7wfmw\""
Jul 23 17:33:58.187797 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:33:58.187692    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ab1734b0-svdrx"]
Jul 23 17:33:58.358233 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:33:58.358199    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c93c6c0d-62f6-42e0-a5af-5a635eb26c91-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-ab1734b0-svdrx\" (UID: \"c93c6c0d-62f6-42e0-a5af-5a635eb26c91\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ab1734b0-svdrx"
Jul 23 17:33:58.458969 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:33:58.458932    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c93c6c0d-62f6-42e0-a5af-5a635eb26c91-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-ab1734b0-svdrx\" (UID: \"c93c6c0d-62f6-42e0-a5af-5a635eb26c91\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ab1734b0-svdrx"
Jul 23 17:33:58.459589 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:33:58.459569    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c93c6c0d-62f6-42e0-a5af-5a635eb26c91-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-ab1734b0-svdrx\" (UID: \"c93c6c0d-62f6-42e0-a5af-5a635eb26c91\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ab1734b0-svdrx"
Jul 23 17:33:58.491947 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:33:58.491917    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ab1734b0-svdrx"
Jul 23 17:33:58.625392 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:33:58.625340    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ab1734b0-svdrx"]
Jul 23 17:33:58.626594 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:33:58.626562    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc93c6c0d_62f6_42e0_a5af_5a635eb26c91.slice/crio-d5ef1bab2f10e15181ca20cb84b87ffb6513623ccd8d75eaf406329c15907214 WatchSource:0}: Error finding container d5ef1bab2f10e15181ca20cb84b87ffb6513623ccd8d75eaf406329c15907214: Status 404 returned error can't find the container with id d5ef1bab2f10e15181ca20cb84b87ffb6513623ccd8d75eaf406329c15907214
Jul 23 17:33:58.969659 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:33:58.969624    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ab1734b0-svdrx" event={"ID":"c93c6c0d-62f6-42e0-a5af-5a635eb26c91","Type":"ContainerStarted","Data":"d5ef1bab2f10e15181ca20cb84b87ffb6513623ccd8d75eaf406329c15907214"}
Jul 23 17:34:01.988096 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:34:01.988058    2577 generic.go:356] "Generic (PLEG): container finished" podID="c93c6c0d-62f6-42e0-a5af-5a635eb26c91" containerID="f6321398abb60bf228cd37c47e83ea29ec2a0e5ed9918a16604fe48885aaec2e" exitCode=0
Jul 23 17:34:01.988535 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:34:01.988148    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ab1734b0-svdrx" event={"ID":"c93c6c0d-62f6-42e0-a5af-5a635eb26c91","Type":"ContainerDied","Data":"f6321398abb60bf228cd37c47e83ea29ec2a0e5ed9918a16604fe48885aaec2e"}
Jul 23 17:34:03.128656 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:34:03.128633    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ab1734b0-svdrx"
Jul 23 17:34:03.201101 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:34:03.201075    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c93c6c0d-62f6-42e0-a5af-5a635eb26c91-maas-api-service-ca\") pod \"c93c6c0d-62f6-42e0-a5af-5a635eb26c91\" (UID: \"c93c6c0d-62f6-42e0-a5af-5a635eb26c91\") "
Jul 23 17:34:03.201443 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:34:03.201421    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c93c6c0d-62f6-42e0-a5af-5a635eb26c91-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c93c6c0d-62f6-42e0-a5af-5a635eb26c91" (UID: "c93c6c0d-62f6-42e0-a5af-5a635eb26c91"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 17:34:03.301780 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:34:03.301702    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c93c6c0d-62f6-42e0-a5af-5a635eb26c91-maas-api-service-ca\") on node \"ip-10-0-129-216.ec2.internal\" DevicePath \"\""
Jul 23 17:34:04.000320 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:34:04.000287    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ab1734b0-svdrx" event={"ID":"c93c6c0d-62f6-42e0-a5af-5a635eb26c91","Type":"ContainerDied","Data":"d5ef1bab2f10e15181ca20cb84b87ffb6513623ccd8d75eaf406329c15907214"}
Jul 23 17:34:04.000320 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:34:04.000303    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ab1734b0-svdrx"
Jul 23 17:34:04.000534 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:34:04.000336    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d5ef1bab2f10e15181ca20cb84b87ffb6513623ccd8d75eaf406329c15907214"
Jul 23 17:34:07.787099 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:34:07.787044    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:34:07.887721 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:34:07.887690    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:34:08.721402 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:34:08.721350    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-ab1734b0-59c5888b6f-zvgbd"]
Jul 23 17:34:08.721679 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:34:08.721612    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ab1734b0-59c5888b6f-zvgbd" podUID="cd76e99a-e2af-46fb-9165-63d019330bc5" containerName="maas-api" containerID="cri-o://ec55f00d537201573476bd9d9f9593aee3840915c1c78f298386c8dcc56e820b" gracePeriod=30
Jul 23 17:34:08.753451 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:34:08.753427    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:34:08.979587 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:34:08.979533    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ab1734b0-59c5888b6f-zvgbd"
Jul 23 17:34:09.026593 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:34:09.026561    2577 generic.go:356] "Generic (PLEG): container finished" podID="cd76e99a-e2af-46fb-9165-63d019330bc5" containerID="ec55f00d537201573476bd9d9f9593aee3840915c1c78f298386c8dcc56e820b" exitCode=0
Jul 23 17:34:09.026785 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:34:09.026609    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ab1734b0-59c5888b6f-zvgbd"
Jul 23 17:34:09.026785 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:34:09.026646    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ab1734b0-59c5888b6f-zvgbd" event={"ID":"cd76e99a-e2af-46fb-9165-63d019330bc5","Type":"ContainerDied","Data":"ec55f00d537201573476bd9d9f9593aee3840915c1c78f298386c8dcc56e820b"}
Jul 23 17:34:09.026785 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:34:09.026700    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ab1734b0-59c5888b6f-zvgbd" event={"ID":"cd76e99a-e2af-46fb-9165-63d019330bc5","Type":"ContainerDied","Data":"1153666ba853fd43179578008798436675c56695855ab8cbc1fc0e2a63555db3"}
Jul 23 17:34:09.026785 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:34:09.026727    2577 scope.go:117] "RemoveContainer" containerID="ec55f00d537201573476bd9d9f9593aee3840915c1c78f298386c8dcc56e820b"
Jul 23 17:34:09.035764 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:34:09.035743    2577 scope.go:117] "RemoveContainer" containerID="ec55f00d537201573476bd9d9f9593aee3840915c1c78f298386c8dcc56e820b"
Jul 23 17:34:09.036036 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:34:09.036011    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec55f00d537201573476bd9d9f9593aee3840915c1c78f298386c8dcc56e820b\": container with ID starting with ec55f00d537201573476bd9d9f9593aee3840915c1c78f298386c8dcc56e820b not found: ID does not exist" containerID="ec55f00d537201573476bd9d9f9593aee3840915c1c78f298386c8dcc56e820b"
Jul 23 17:34:09.036117 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:34:09.036048    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec55f00d537201573476bd9d9f9593aee3840915c1c78f298386c8dcc56e820b"} err="failed to get container status \"ec55f00d537201573476bd9d9f9593aee3840915c1c78f298386c8dcc56e820b\": rpc error: code = NotFound desc = could not find container \"ec55f00d537201573476bd9d9f9593aee3840915c1c78f298386c8dcc56e820b\": container with ID starting with ec55f00d537201573476bd9d9f9593aee3840915c1c78f298386c8dcc56e820b not found: ID does not exist"
Jul 23 17:34:09.048495 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:34:09.048475    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd76e99a-e2af-46fb-9165-63d019330bc5-openshift-service-ca\") pod \"cd76e99a-e2af-46fb-9165-63d019330bc5\" (UID: \"cd76e99a-e2af-46fb-9165-63d019330bc5\") "
Jul 23 17:34:09.048617 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:34:09.048524    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cd76e99a-e2af-46fb-9165-63d019330bc5-maas-api-tls\") pod \"cd76e99a-e2af-46fb-9165-63d019330bc5\" (UID: \"cd76e99a-e2af-46fb-9165-63d019330bc5\") "
Jul 23 17:34:09.048617 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:34:09.048555    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p2lxz\" (UniqueName: \"kubernetes.io/projected/cd76e99a-e2af-46fb-9165-63d019330bc5-kube-api-access-p2lxz\") pod \"cd76e99a-e2af-46fb-9165-63d019330bc5\" (UID: \"cd76e99a-e2af-46fb-9165-63d019330bc5\") "
Jul 23 17:34:09.048973 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:34:09.048938    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd76e99a-e2af-46fb-9165-63d019330bc5-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "cd76e99a-e2af-46fb-9165-63d019330bc5" (UID: "cd76e99a-e2af-46fb-9165-63d019330bc5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 17:34:09.050627 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:34:09.050600    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd76e99a-e2af-46fb-9165-63d019330bc5-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "cd76e99a-e2af-46fb-9165-63d019330bc5" (UID: "cd76e99a-e2af-46fb-9165-63d019330bc5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 17:34:09.050764 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:34:09.050742    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd76e99a-e2af-46fb-9165-63d019330bc5-kube-api-access-p2lxz" (OuterVolumeSpecName: "kube-api-access-p2lxz") pod "cd76e99a-e2af-46fb-9165-63d019330bc5" (UID: "cd76e99a-e2af-46fb-9165-63d019330bc5"). InnerVolumeSpecName "kube-api-access-p2lxz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 17:34:09.149689 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:34:09.149662    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cd76e99a-e2af-46fb-9165-63d019330bc5-maas-api-tls\") on node \"ip-10-0-129-216.ec2.internal\" DevicePath \"\""
Jul 23 17:34:09.149689 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:34:09.149687    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p2lxz\" (UniqueName: \"kubernetes.io/projected/cd76e99a-e2af-46fb-9165-63d019330bc5-kube-api-access-p2lxz\") on node \"ip-10-0-129-216.ec2.internal\" DevicePath \"\""
Jul 23 17:34:09.149856 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:34:09.149698    2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd76e99a-e2af-46fb-9165-63d019330bc5-openshift-service-ca\") on node \"ip-10-0-129-216.ec2.internal\" DevicePath \"\""
Jul 23 17:34:09.351799 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:34:09.351763    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-ab1734b0-59c5888b6f-zvgbd"]
Jul 23 17:34:09.354018 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:34:09.353997    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-ab1734b0-59c5888b6f-zvgbd"]
Jul 23 17:34:10.765915 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:34:10.765883    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cd76e99a-e2af-46fb-9165-63d019330bc5" path="/var/lib/kubelet/pods/cd76e99a-e2af-46fb-9165-63d019330bc5/volumes"
Jul 23 17:34:13.259238 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:34:13.259199    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ab1734b0-svdrx"]
Jul 23 17:34:13.261791 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:34:13.261761    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:34:13.262758 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:34:13.262737    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ab1734b0-svdrx"]
Jul 23 17:34:13.438294 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:34:13.438246    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:34:13.677611 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:34:13.677531    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:34:13.935861 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:34:13.935788    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:34:14.352047 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:34:14.352009    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:34:14.767291 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:34:14.767239    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c93c6c0d-62f6-42e0-a5af-5a635eb26c91" path="/var/lib/kubelet/pods/c93c6c0d-62f6-42e0-a5af-5a635eb26c91/volumes"
Jul 23 17:34:14.794666 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:34:14.794636    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:34:17.253067 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:34:17.253024    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:34:17.342135 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:34:17.342099    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0e2114d4-64fb7f946c-pnngf"]
Jul 23 17:34:17.342555 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:34:17.342540    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c93c6c0d-62f6-42e0-a5af-5a635eb26c91" containerName="revoke-keys"
Jul 23 17:34:17.342555 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:34:17.342557    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="c93c6c0d-62f6-42e0-a5af-5a635eb26c91" containerName="revoke-keys"
Jul 23 17:34:17.342669 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:34:17.342567    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cd76e99a-e2af-46fb-9165-63d019330bc5" containerName="maas-api"
Jul 23 17:34:17.342669 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:34:17.342573    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="cd76e99a-e2af-46fb-9165-63d019330bc5" containerName="maas-api"
Jul 23 17:34:17.342669 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:34:17.342640    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="c93c6c0d-62f6-42e0-a5af-5a635eb26c91" containerName="revoke-keys"
Jul 23 17:34:17.342669 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:34:17.342649    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="cd76e99a-e2af-46fb-9165-63d019330bc5" containerName="maas-api"
Jul 23 17:34:17.346903 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:34:17.346886    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e2114d4-64fb7f946c-pnngf"
Jul 23 17:34:17.348438 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:34:17.348409    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:34:17.350654 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:34:17.350630    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-0e2114d4-serving-cert\""
Jul 23 17:34:17.362445 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:34:17.362422    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0e2114d4-64fb7f946c-pnngf"]
Jul 23 17:34:17.424753 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:34:17.424718    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/91300d30-4e5e-429d-8892-c745b70231e1-maas-api-tls\") pod \"maas-api-e2e-ait-0e2114d4-64fb7f946c-pnngf\" (UID: \"91300d30-4e5e-429d-8892-c745b70231e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e2114d4-64fb7f946c-pnngf"
Jul 23 17:34:17.424906 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:34:17.424838    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/91300d30-4e5e-429d-8892-c745b70231e1-openshift-service-ca\") pod \"maas-api-e2e-ait-0e2114d4-64fb7f946c-pnngf\" (UID: \"91300d30-4e5e-429d-8892-c745b70231e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e2114d4-64fb7f946c-pnngf"
Jul 23 17:34:17.424906 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:34:17.424885    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5m5z\" (UniqueName: \"kubernetes.io/projected/91300d30-4e5e-429d-8892-c745b70231e1-kube-api-access-z5m5z\") pod \"maas-api-e2e-ait-0e2114d4-64fb7f946c-pnngf\" (UID: \"91300d30-4e5e-429d-8892-c745b70231e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e2114d4-64fb7f946c-pnngf"
Jul 23 17:34:17.525679 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:34:17.525597    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/91300d30-4e5e-429d-8892-c745b70231e1-openshift-service-ca\") pod \"maas-api-e2e-ait-0e2114d4-64fb7f946c-pnngf\" (UID: \"91300d30-4e5e-429d-8892-c745b70231e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e2114d4-64fb7f946c-pnngf"
Jul 23 17:34:17.525679 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:34:17.525638    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5m5z\" (UniqueName: \"kubernetes.io/projected/91300d30-4e5e-429d-8892-c745b70231e1-kube-api-access-z5m5z\") pod \"maas-api-e2e-ait-0e2114d4-64fb7f946c-pnngf\" (UID: \"91300d30-4e5e-429d-8892-c745b70231e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e2114d4-64fb7f946c-pnngf"
Jul 23 17:34:17.525679 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:34:17.525675    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/91300d30-4e5e-429d-8892-c745b70231e1-maas-api-tls\") pod \"maas-api-e2e-ait-0e2114d4-64fb7f946c-pnngf\" (UID: \"91300d30-4e5e-429d-8892-c745b70231e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e2114d4-64fb7f946c-pnngf"
Jul 23 17:34:17.526202 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:34:17.526181    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/91300d30-4e5e-429d-8892-c745b70231e1-openshift-service-ca\") pod \"maas-api-e2e-ait-0e2114d4-64fb7f946c-pnngf\" (UID: \"91300d30-4e5e-429d-8892-c745b70231e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e2114d4-64fb7f946c-pnngf"
Jul 23 17:34:17.528013 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:34:17.527994    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/91300d30-4e5e-429d-8892-c745b70231e1-maas-api-tls\") pod \"maas-api-e2e-ait-0e2114d4-64fb7f946c-pnngf\" (UID: \"91300d30-4e5e-429d-8892-c745b70231e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e2114d4-64fb7f946c-pnngf"
Jul 23 17:34:17.533880 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:34:17.533856    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5m5z\" (UniqueName: \"kubernetes.io/projected/91300d30-4e5e-429d-8892-c745b70231e1-kube-api-access-z5m5z\") pod \"maas-api-e2e-ait-0e2114d4-64fb7f946c-pnngf\" (UID: \"91300d30-4e5e-429d-8892-c745b70231e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e2114d4-64fb7f946c-pnngf"
Jul 23 17:34:17.608892 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:34:17.608854    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:34:17.660300 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:34:17.660243    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e2114d4-64fb7f946c-pnngf"
Jul 23 17:34:17.801639 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:34:17.801613    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0e2114d4-64fb7f946c-pnngf"]
Jul 23 17:34:17.803208 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:34:17.803184    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91300d30_4e5e_429d_8892_c745b70231e1.slice/crio-cdab2f88c364f6e2f7e6df99c383b942d7cfae7521863ee0b9419aded969e06d WatchSource:0}: Error finding container cdab2f88c364f6e2f7e6df99c383b942d7cfae7521863ee0b9419aded969e06d: Status 404 returned error can't find the container with id cdab2f88c364f6e2f7e6df99c383b942d7cfae7521863ee0b9419aded969e06d
Jul 23 17:34:17.903407 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:34:17.903375    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:34:18.061719 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:34:18.061609    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:34:18.077777 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:34:18.077749    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e2114d4-64fb7f946c-pnngf" event={"ID":"91300d30-4e5e-429d-8892-c745b70231e1","Type":"ContainerStarted","Data":"cdab2f88c364f6e2f7e6df99c383b942d7cfae7521863ee0b9419aded969e06d"}
Jul 23 17:34:18.166052 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:34:18.166020    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:34:19.085059 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:34:19.085026    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e2114d4-64fb7f946c-pnngf" event={"ID":"91300d30-4e5e-429d-8892-c745b70231e1","Type":"ContainerStarted","Data":"df3332b326f457133332f55d6dbb2ef9ee4f951f7738779b48fc2dc7ab5dadf0"}
Jul 23 17:34:19.085581 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:34:19.085143    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e2114d4-64fb7f946c-pnngf"
Jul 23 17:34:19.104234 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:34:19.104184    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e2114d4-64fb7f946c-pnngf" podStartSLOduration=1.658138912 podStartE2EDuration="2.10416695s" totalImagesPullingTime="446.028038ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 17:34:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 17:34:19.102742115 +0000 UTC m=+1754.873027670" watchObservedRunningTime="2026-07-23 17:34:19.10416695 +0000 UTC m=+1754.874452468"
Jul 23 17:34:23.385902 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:34:23.385872    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:34:23.446720 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:34:23.446687    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:34:25.098724 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:34:25.098689    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e2114d4-64fb7f946c-pnngf"
Jul 23 17:34:41.808967 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:34:41.808930    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:34:43.324446 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:34:43.324404    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:34:43.424933 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:34:43.424902    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:34:44.384054 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:34:44.384023    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:34:44.479785 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:34:44.479756    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:34:45.246932 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:34:45.246899    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:34:45.347680 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:34:45.347649    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:34:46.341831 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:34:46.341799    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:34:46.445275 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:34:46.445223    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:34:47.138928 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:34:47.138899    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:34:47.239053 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:34:47.239005    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:34:48.162610 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:34:48.162574    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:34:48.261469 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:34:48.261435    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:34:49.070050 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:34:49.070015    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:34:49.168741 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:34:49.168705    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:34:49.861245 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:34:49.861211    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:34:49.962195 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:34:49.962160    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:34:50.690485 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:34:50.690455    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:34:50.790797 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:34:50.790765    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:34:51.715475 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:34:51.715443    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:34:51.816928 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:34:51.816894    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:34:52.526820 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:34:52.526783    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:34:52.624435 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:34:52.624405    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:34:53.546681 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:34:53.546643    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:34:53.647114 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:34:53.647079    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:34:54.396640 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:34:54.396607    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:34:54.497674 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:34:54.497639    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:34:54.662645 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:34:54.662549    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0e2114d4-64fb7f946c-pnngf"]
Jul 23 17:34:54.663082 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:34:54.662795    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e2114d4-64fb7f946c-pnngf" podUID="91300d30-4e5e-429d-8892-c745b70231e1" containerName="maas-api" containerID="cri-o://df3332b326f457133332f55d6dbb2ef9ee4f951f7738779b48fc2dc7ab5dadf0" gracePeriod=30
Jul 23 17:34:54.680642 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:34:54.680598    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:34:54.921758 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:34:54.921704    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e2114d4-64fb7f946c-pnngf"
Jul 23 17:34:54.961559 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:34:54.961531    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/91300d30-4e5e-429d-8892-c745b70231e1-maas-api-tls\") pod \"91300d30-4e5e-429d-8892-c745b70231e1\" (UID: \"91300d30-4e5e-429d-8892-c745b70231e1\") "
Jul 23 17:34:54.961721 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:34:54.961588    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z5m5z\" (UniqueName: \"kubernetes.io/projected/91300d30-4e5e-429d-8892-c745b70231e1-kube-api-access-z5m5z\") pod \"91300d30-4e5e-429d-8892-c745b70231e1\" (UID: \"91300d30-4e5e-429d-8892-c745b70231e1\") "
Jul 23 17:34:54.961721 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:34:54.961627    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/91300d30-4e5e-429d-8892-c745b70231e1-openshift-service-ca\") pod \"91300d30-4e5e-429d-8892-c745b70231e1\" (UID: \"91300d30-4e5e-429d-8892-c745b70231e1\") "
Jul 23 17:34:54.962001 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:34:54.961975    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91300d30-4e5e-429d-8892-c745b70231e1-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "91300d30-4e5e-429d-8892-c745b70231e1" (UID: "91300d30-4e5e-429d-8892-c745b70231e1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 17:34:54.963554 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:34:54.963531    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91300d30-4e5e-429d-8892-c745b70231e1-kube-api-access-z5m5z" (OuterVolumeSpecName: "kube-api-access-z5m5z") pod "91300d30-4e5e-429d-8892-c745b70231e1" (UID: "91300d30-4e5e-429d-8892-c745b70231e1"). InnerVolumeSpecName "kube-api-access-z5m5z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 17:34:54.963627 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:34:54.963610    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/91300d30-4e5e-429d-8892-c745b70231e1-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "91300d30-4e5e-429d-8892-c745b70231e1" (UID: "91300d30-4e5e-429d-8892-c745b70231e1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 17:34:55.063243 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:34:55.063210    2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/91300d30-4e5e-429d-8892-c745b70231e1-openshift-service-ca\") on node \"ip-10-0-129-216.ec2.internal\" DevicePath \"\""
Jul 23 17:34:55.063243 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:34:55.063239    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/91300d30-4e5e-429d-8892-c745b70231e1-maas-api-tls\") on node \"ip-10-0-129-216.ec2.internal\" DevicePath \"\""
Jul 23 17:34:55.063243 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:34:55.063249    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z5m5z\" (UniqueName: \"kubernetes.io/projected/91300d30-4e5e-429d-8892-c745b70231e1-kube-api-access-z5m5z\") on node \"ip-10-0-129-216.ec2.internal\" DevicePath \"\""
Jul 23 17:34:55.289547 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:34:55.289513    2577 generic.go:356] "Generic (PLEG): container finished" podID="91300d30-4e5e-429d-8892-c745b70231e1" containerID="df3332b326f457133332f55d6dbb2ef9ee4f951f7738779b48fc2dc7ab5dadf0" exitCode=0
Jul 23 17:34:55.289744 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:34:55.289560    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e2114d4-64fb7f946c-pnngf"
Jul 23 17:34:55.289744 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:34:55.289585    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e2114d4-64fb7f946c-pnngf" event={"ID":"91300d30-4e5e-429d-8892-c745b70231e1","Type":"ContainerDied","Data":"df3332b326f457133332f55d6dbb2ef9ee4f951f7738779b48fc2dc7ab5dadf0"}
Jul 23 17:34:55.289744 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:34:55.289628    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e2114d4-64fb7f946c-pnngf" event={"ID":"91300d30-4e5e-429d-8892-c745b70231e1","Type":"ContainerDied","Data":"cdab2f88c364f6e2f7e6df99c383b942d7cfae7521863ee0b9419aded969e06d"}
Jul 23 17:34:55.289744 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:34:55.289652    2577 scope.go:117] "RemoveContainer" containerID="df3332b326f457133332f55d6dbb2ef9ee4f951f7738779b48fc2dc7ab5dadf0"
Jul 23 17:34:55.299504 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:34:55.299485    2577 scope.go:117] "RemoveContainer" containerID="df3332b326f457133332f55d6dbb2ef9ee4f951f7738779b48fc2dc7ab5dadf0"
Jul 23 17:34:55.299746 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:34:55.299729    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df3332b326f457133332f55d6dbb2ef9ee4f951f7738779b48fc2dc7ab5dadf0\": container with ID starting with df3332b326f457133332f55d6dbb2ef9ee4f951f7738779b48fc2dc7ab5dadf0 not found: ID does not exist" containerID="df3332b326f457133332f55d6dbb2ef9ee4f951f7738779b48fc2dc7ab5dadf0"
Jul 23 17:34:55.299798 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:34:55.299758    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df3332b326f457133332f55d6dbb2ef9ee4f951f7738779b48fc2dc7ab5dadf0"} err="failed to get container status \"df3332b326f457133332f55d6dbb2ef9ee4f951f7738779b48fc2dc7ab5dadf0\": rpc error: code = NotFound desc = could not find container \"df3332b326f457133332f55d6dbb2ef9ee4f951f7738779b48fc2dc7ab5dadf0\": container with ID starting with df3332b326f457133332f55d6dbb2ef9ee4f951f7738779b48fc2dc7ab5dadf0 not found: ID does not exist"
Jul 23 17:34:55.325538 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:34:55.325505    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0e2114d4-64fb7f946c-pnngf"]
Jul 23 17:34:55.336721 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:34:55.336692    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0e2114d4-64fb7f946c-pnngf"]
Jul 23 17:34:56.766306 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:34:56.766242    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="91300d30-4e5e-429d-8892-c745b70231e1" path="/var/lib/kubelet/pods/91300d30-4e5e-429d-8892-c745b70231e1/volumes"
Jul 23 17:34:58.203484 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:34:58.203447    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:34:59.530838 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:34:59.530804    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:34:59.735566 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:34:59.735531    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:34:59.988527 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:34:59.988493    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:35:00.371707 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:35:00.371596    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:35:00.683658 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:35:00.683576    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:35:02.500068 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:35:02.500015    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:35:02.606325 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:35:02.606284    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-e6ac8a29-564695df4d-jgfwq"]
Jul 23 17:35:02.606704 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:35:02.606691    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="91300d30-4e5e-429d-8892-c745b70231e1" containerName="maas-api"
Jul 23 17:35:02.606776 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:35:02.606706    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="91300d30-4e5e-429d-8892-c745b70231e1" containerName="maas-api"
Jul 23 17:35:02.606834 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:35:02.606787    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="91300d30-4e5e-429d-8892-c745b70231e1" containerName="maas-api"
Jul 23 17:35:02.608836 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:35:02.608815    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e6ac8a29-564695df4d-jgfwq"
Jul 23 17:35:02.611444 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:35:02.611426    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-e6ac8a29-serving-cert\""
Jul 23 17:35:02.621769 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:35:02.621744    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-e6ac8a29-564695df4d-jgfwq"]
Jul 23 17:35:02.633195 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:35:02.633112    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fdf9d680-1864-4545-ac50-7b260943faa6-maas-api-tls\") pod \"maas-api-e2e-derive-e6ac8a29-564695df4d-jgfwq\" (UID: \"fdf9d680-1864-4545-ac50-7b260943faa6\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e6ac8a29-564695df4d-jgfwq"
Jul 23 17:35:02.633368 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:35:02.633288    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fdf9d680-1864-4545-ac50-7b260943faa6-openshift-service-ca\") pod \"maas-api-e2e-derive-e6ac8a29-564695df4d-jgfwq\" (UID: \"fdf9d680-1864-4545-ac50-7b260943faa6\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e6ac8a29-564695df4d-jgfwq"
Jul 23 17:35:02.633368 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:35:02.633339    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4n4zx\" (UniqueName: \"kubernetes.io/projected/fdf9d680-1864-4545-ac50-7b260943faa6-kube-api-access-4n4zx\") pod \"maas-api-e2e-derive-e6ac8a29-564695df4d-jgfwq\" (UID: \"fdf9d680-1864-4545-ac50-7b260943faa6\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e6ac8a29-564695df4d-jgfwq"
Jul 23 17:35:02.651566 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:35:02.651537    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:35:02.734352 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:35:02.734319    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4n4zx\" (UniqueName: \"kubernetes.io/projected/fdf9d680-1864-4545-ac50-7b260943faa6-kube-api-access-4n4zx\") pod \"maas-api-e2e-derive-e6ac8a29-564695df4d-jgfwq\" (UID: \"fdf9d680-1864-4545-ac50-7b260943faa6\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e6ac8a29-564695df4d-jgfwq"
Jul 23 17:35:02.734570 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:35:02.734365    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fdf9d680-1864-4545-ac50-7b260943faa6-maas-api-tls\") pod \"maas-api-e2e-derive-e6ac8a29-564695df4d-jgfwq\" (UID: \"fdf9d680-1864-4545-ac50-7b260943faa6\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e6ac8a29-564695df4d-jgfwq"
Jul 23 17:35:02.734570 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:35:02.734438    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fdf9d680-1864-4545-ac50-7b260943faa6-openshift-service-ca\") pod \"maas-api-e2e-derive-e6ac8a29-564695df4d-jgfwq\" (UID: \"fdf9d680-1864-4545-ac50-7b260943faa6\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e6ac8a29-564695df4d-jgfwq"
Jul 23 17:35:02.734570 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:35:02.734548    2577 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-e6ac8a29-serving-cert: secret "maas-api-e2e-derive-e6ac8a29-serving-cert" not found
Jul 23 17:35:02.734804 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:35:02.734615    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fdf9d680-1864-4545-ac50-7b260943faa6-maas-api-tls podName:fdf9d680-1864-4545-ac50-7b260943faa6 nodeName:}" failed. No retries permitted until 2026-07-23 17:35:03.234597883 +0000 UTC m=+1799.004883381 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/fdf9d680-1864-4545-ac50-7b260943faa6-maas-api-tls") pod "maas-api-e2e-derive-e6ac8a29-564695df4d-jgfwq" (UID: "fdf9d680-1864-4545-ac50-7b260943faa6") : secret "maas-api-e2e-derive-e6ac8a29-serving-cert" not found
Jul 23 17:35:02.735007 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:35:02.734988    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fdf9d680-1864-4545-ac50-7b260943faa6-openshift-service-ca\") pod \"maas-api-e2e-derive-e6ac8a29-564695df4d-jgfwq\" (UID: \"fdf9d680-1864-4545-ac50-7b260943faa6\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e6ac8a29-564695df4d-jgfwq"
Jul 23 17:35:02.742854 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:35:02.742822    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4n4zx\" (UniqueName: \"kubernetes.io/projected/fdf9d680-1864-4545-ac50-7b260943faa6-kube-api-access-4n4zx\") pod \"maas-api-e2e-derive-e6ac8a29-564695df4d-jgfwq\" (UID: \"fdf9d680-1864-4545-ac50-7b260943faa6\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e6ac8a29-564695df4d-jgfwq"
Jul 23 17:35:02.756493 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:35:02.756465    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:35:03.167561 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:35:03.167526    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:35:03.239393 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:35:03.239360    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fdf9d680-1864-4545-ac50-7b260943faa6-maas-api-tls\") pod \"maas-api-e2e-derive-e6ac8a29-564695df4d-jgfwq\" (UID: \"fdf9d680-1864-4545-ac50-7b260943faa6\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e6ac8a29-564695df4d-jgfwq"
Jul 23 17:35:03.239558 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:35:03.239518    2577 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-e6ac8a29-serving-cert: secret "maas-api-e2e-derive-e6ac8a29-serving-cert" not found
Jul 23 17:35:03.239613 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:35:03.239584    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fdf9d680-1864-4545-ac50-7b260943faa6-maas-api-tls podName:fdf9d680-1864-4545-ac50-7b260943faa6 nodeName:}" failed. No retries permitted until 2026-07-23 17:35:04.239570098 +0000 UTC m=+1800.009855595 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/fdf9d680-1864-4545-ac50-7b260943faa6-maas-api-tls") pod "maas-api-e2e-derive-e6ac8a29-564695df4d-jgfwq" (UID: "fdf9d680-1864-4545-ac50-7b260943faa6") : secret "maas-api-e2e-derive-e6ac8a29-serving-cert" not found
Jul 23 17:35:03.267646 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:35:03.267611    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:35:03.367793 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:35:03.367764    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:35:04.250091 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:35:04.250050    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fdf9d680-1864-4545-ac50-7b260943faa6-maas-api-tls\") pod \"maas-api-e2e-derive-e6ac8a29-564695df4d-jgfwq\" (UID: \"fdf9d680-1864-4545-ac50-7b260943faa6\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e6ac8a29-564695df4d-jgfwq"
Jul 23 17:35:04.252473 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:35:04.252442    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fdf9d680-1864-4545-ac50-7b260943faa6-maas-api-tls\") pod \"maas-api-e2e-derive-e6ac8a29-564695df4d-jgfwq\" (UID: \"fdf9d680-1864-4545-ac50-7b260943faa6\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e6ac8a29-564695df4d-jgfwq"
Jul 23 17:35:04.419167 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:35:04.419134    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e6ac8a29-564695df4d-jgfwq"
Jul 23 17:35:04.556531 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:35:04.556506    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-e6ac8a29-564695df4d-jgfwq"]
Jul 23 17:35:04.558236 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:35:04.558212    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdf9d680_1864_4545_ac50_7b260943faa6.slice/crio-835c1084b84ef09f7c34c5ae18d1382fb7e2ac65ba3698f37b470718dd236c69 WatchSource:0}: Error finding container 835c1084b84ef09f7c34c5ae18d1382fb7e2ac65ba3698f37b470718dd236c69: Status 404 returned error can't find the container with id 835c1084b84ef09f7c34c5ae18d1382fb7e2ac65ba3698f37b470718dd236c69
Jul 23 17:35:05.352418 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:35:05.352381    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e6ac8a29-564695df4d-jgfwq" event={"ID":"fdf9d680-1864-4545-ac50-7b260943faa6","Type":"ContainerStarted","Data":"5123569db3102341fb9c53576d6ae6641bc3b28d24dd854ff956317cf7f27f85"}
Jul 23 17:35:05.352797 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:35:05.352433    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e6ac8a29-564695df4d-jgfwq" event={"ID":"fdf9d680-1864-4545-ac50-7b260943faa6","Type":"ContainerStarted","Data":"835c1084b84ef09f7c34c5ae18d1382fb7e2ac65ba3698f37b470718dd236c69"}
Jul 23 17:35:05.352797 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:35:05.352464    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e6ac8a29-564695df4d-jgfwq"
Jul 23 17:35:05.371728 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:35:05.371678    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e6ac8a29-564695df4d-jgfwq" podStartSLOduration=2.7665777499999997 podStartE2EDuration="3.371661099s" totalImagesPullingTime="605.083349ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 17:35:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 17:35:05.369695749 +0000 UTC m=+1801.139981267" watchObservedRunningTime="2026-07-23 17:35:05.371661099 +0000 UTC m=+1801.141946618"
Jul 23 17:35:11.363028 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:35:11.363000    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e6ac8a29-564695df4d-jgfwq"
Jul 23 17:35:27.423908 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:35:27.423871    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e6ac8a29-db5wf"]
Jul 23 17:35:27.426436 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:35:27.426418    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e6ac8a29-db5wf"
Jul 23 17:35:27.428589 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:35:27.428563    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7wfmw\""
Jul 23 17:35:27.434713 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:35:27.434447    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e6ac8a29-db5wf"]
Jul 23 17:35:27.556083 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:35:27.556040    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c14974c-4d7d-47e0-bdc6-a896d1d4918b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-e6ac8a29-db5wf\" (UID: \"9c14974c-4d7d-47e0-bdc6-a896d1d4918b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e6ac8a29-db5wf"
Jul 23 17:35:27.657558 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:35:27.657525    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c14974c-4d7d-47e0-bdc6-a896d1d4918b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-e6ac8a29-db5wf\" (UID: \"9c14974c-4d7d-47e0-bdc6-a896d1d4918b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e6ac8a29-db5wf"
Jul 23 17:35:27.658089 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:35:27.658069    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c14974c-4d7d-47e0-bdc6-a896d1d4918b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-e6ac8a29-db5wf\" (UID: \"9c14974c-4d7d-47e0-bdc6-a896d1d4918b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e6ac8a29-db5wf"
Jul 23 17:35:27.739317 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:35:27.739292    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e6ac8a29-db5wf"
Jul 23 17:35:27.877732 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:35:27.877709    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e6ac8a29-db5wf"]
Jul 23 17:35:27.879436 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:35:27.879401    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c14974c_4d7d_47e0_bdc6_a896d1d4918b.slice/crio-b97690a021d8ad0f67574563fc4548c5efb19d03097f3466a110dad1a513961f WatchSource:0}: Error finding container b97690a021d8ad0f67574563fc4548c5efb19d03097f3466a110dad1a513961f: Status 404 returned error can't find the container with id b97690a021d8ad0f67574563fc4548c5efb19d03097f3466a110dad1a513961f
Jul 23 17:35:28.481341 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:35:28.481304    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e6ac8a29-db5wf" event={"ID":"9c14974c-4d7d-47e0-bdc6-a896d1d4918b","Type":"ContainerStarted","Data":"5e9089d0b3c360d28d7d3420648d870a111489f9bcd820108be5eca771300ede"}
Jul 23 17:35:28.481714 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:35:28.481370    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e6ac8a29-db5wf" event={"ID":"9c14974c-4d7d-47e0-bdc6-a896d1d4918b","Type":"ContainerStarted","Data":"b97690a021d8ad0f67574563fc4548c5efb19d03097f3466a110dad1a513961f"}
Jul 23 17:35:28.497790 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:35:28.497743    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e6ac8a29-db5wf" podStartSLOduration=1.497726746 podStartE2EDuration="1.497726746s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 17:35:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 17:35:28.495618793 +0000 UTC m=+1824.265904313" watchObservedRunningTime="2026-07-23 17:35:28.497726746 +0000 UTC m=+1824.268012264"
Jul 23 17:35:29.492498 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:35:29.492461    2577 generic.go:356] "Generic (PLEG): container finished" podID="9c14974c-4d7d-47e0-bdc6-a896d1d4918b" containerID="5e9089d0b3c360d28d7d3420648d870a111489f9bcd820108be5eca771300ede" exitCode=0
Jul 23 17:35:29.492900 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:35:29.492542    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e6ac8a29-db5wf" event={"ID":"9c14974c-4d7d-47e0-bdc6-a896d1d4918b","Type":"ContainerDied","Data":"5e9089d0b3c360d28d7d3420648d870a111489f9bcd820108be5eca771300ede"}
Jul 23 17:35:30.630517 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:35:30.630484    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e6ac8a29-db5wf"
Jul 23 17:35:30.785652 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:35:30.785579    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c14974c-4d7d-47e0-bdc6-a896d1d4918b-maas-api-service-ca\") pod \"9c14974c-4d7d-47e0-bdc6-a896d1d4918b\" (UID: \"9c14974c-4d7d-47e0-bdc6-a896d1d4918b\") "
Jul 23 17:35:30.785922 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:35:30.785896    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c14974c-4d7d-47e0-bdc6-a896d1d4918b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "9c14974c-4d7d-47e0-bdc6-a896d1d4918b" (UID: "9c14974c-4d7d-47e0-bdc6-a896d1d4918b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 17:35:30.887189 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:35:30.887154    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c14974c-4d7d-47e0-bdc6-a896d1d4918b-maas-api-service-ca\") on node \"ip-10-0-129-216.ec2.internal\" DevicePath \"\""
Jul 23 17:35:31.504084 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:35:31.504054    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e6ac8a29-db5wf"
Jul 23 17:35:31.504288 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:35:31.504095    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e6ac8a29-db5wf" event={"ID":"9c14974c-4d7d-47e0-bdc6-a896d1d4918b","Type":"ContainerDied","Data":"b97690a021d8ad0f67574563fc4548c5efb19d03097f3466a110dad1a513961f"}
Jul 23 17:35:31.504288 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:35:31.504142    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b97690a021d8ad0f67574563fc4548c5efb19d03097f3466a110dad1a513961f"
Jul 23 17:35:38.016617 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:35:38.016579    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-e6ac8a29-564695df4d-jgfwq"]
Jul 23 17:35:38.017104 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:35:38.016774    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e6ac8a29-564695df4d-jgfwq" podUID="fdf9d680-1864-4545-ac50-7b260943faa6" containerName="maas-api" containerID="cri-o://5123569db3102341fb9c53576d6ae6641bc3b28d24dd854ff956317cf7f27f85" gracePeriod=30
Jul 23 17:35:38.042838 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:35:38.042652    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:35:38.271346 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:35:38.271290    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e6ac8a29-564695df4d-jgfwq"
Jul 23 17:35:38.454355 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:35:38.454326    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fdf9d680-1864-4545-ac50-7b260943faa6-openshift-service-ca\") pod \"fdf9d680-1864-4545-ac50-7b260943faa6\" (UID: \"fdf9d680-1864-4545-ac50-7b260943faa6\") "
Jul 23 17:35:38.454522 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:35:38.454434    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4n4zx\" (UniqueName: \"kubernetes.io/projected/fdf9d680-1864-4545-ac50-7b260943faa6-kube-api-access-4n4zx\") pod \"fdf9d680-1864-4545-ac50-7b260943faa6\" (UID: \"fdf9d680-1864-4545-ac50-7b260943faa6\") "
Jul 23 17:35:38.454522 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:35:38.454458    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fdf9d680-1864-4545-ac50-7b260943faa6-maas-api-tls\") pod \"fdf9d680-1864-4545-ac50-7b260943faa6\" (UID: \"fdf9d680-1864-4545-ac50-7b260943faa6\") "
Jul 23 17:35:38.454747 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:35:38.454717    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fdf9d680-1864-4545-ac50-7b260943faa6-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "fdf9d680-1864-4545-ac50-7b260943faa6" (UID: "fdf9d680-1864-4545-ac50-7b260943faa6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 17:35:38.456439 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:35:38.456411    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fdf9d680-1864-4545-ac50-7b260943faa6-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "fdf9d680-1864-4545-ac50-7b260943faa6" (UID: "fdf9d680-1864-4545-ac50-7b260943faa6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 17:35:38.456578 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:35:38.456553    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdf9d680-1864-4545-ac50-7b260943faa6-kube-api-access-4n4zx" (OuterVolumeSpecName: "kube-api-access-4n4zx") pod "fdf9d680-1864-4545-ac50-7b260943faa6" (UID: "fdf9d680-1864-4545-ac50-7b260943faa6"). InnerVolumeSpecName "kube-api-access-4n4zx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 17:35:38.544534 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:35:38.544452    2577 generic.go:356] "Generic (PLEG): container finished" podID="fdf9d680-1864-4545-ac50-7b260943faa6" containerID="5123569db3102341fb9c53576d6ae6641bc3b28d24dd854ff956317cf7f27f85" exitCode=0
Jul 23 17:35:38.544534 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:35:38.544500    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e6ac8a29-564695df4d-jgfwq"
Jul 23 17:35:38.544697 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:35:38.544514    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e6ac8a29-564695df4d-jgfwq" event={"ID":"fdf9d680-1864-4545-ac50-7b260943faa6","Type":"ContainerDied","Data":"5123569db3102341fb9c53576d6ae6641bc3b28d24dd854ff956317cf7f27f85"}
Jul 23 17:35:38.544697 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:35:38.544611    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e6ac8a29-564695df4d-jgfwq" event={"ID":"fdf9d680-1864-4545-ac50-7b260943faa6","Type":"ContainerDied","Data":"835c1084b84ef09f7c34c5ae18d1382fb7e2ac65ba3698f37b470718dd236c69"}
Jul 23 17:35:38.544697 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:35:38.544642    2577 scope.go:117] "RemoveContainer" containerID="5123569db3102341fb9c53576d6ae6641bc3b28d24dd854ff956317cf7f27f85"
Jul 23 17:35:38.554299 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:35:38.554269    2577 scope.go:117] "RemoveContainer" containerID="5123569db3102341fb9c53576d6ae6641bc3b28d24dd854ff956317cf7f27f85"
Jul 23 17:35:38.554577 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:35:38.554553    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5123569db3102341fb9c53576d6ae6641bc3b28d24dd854ff956317cf7f27f85\": container with ID starting with 5123569db3102341fb9c53576d6ae6641bc3b28d24dd854ff956317cf7f27f85 not found: ID does not exist" containerID="5123569db3102341fb9c53576d6ae6641bc3b28d24dd854ff956317cf7f27f85"
Jul 23 17:35:38.554647 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:35:38.554591    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5123569db3102341fb9c53576d6ae6641bc3b28d24dd854ff956317cf7f27f85"} err="failed to get container status \"5123569db3102341fb9c53576d6ae6641bc3b28d24dd854ff956317cf7f27f85\": rpc error: code = NotFound desc = could not find container \"5123569db3102341fb9c53576d6ae6641bc3b28d24dd854ff956317cf7f27f85\": container with ID starting with 5123569db3102341fb9c53576d6ae6641bc3b28d24dd854ff956317cf7f27f85 not found: ID does not exist"
Jul 23 17:35:38.555128 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:35:38.555113    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4n4zx\" (UniqueName: \"kubernetes.io/projected/fdf9d680-1864-4545-ac50-7b260943faa6-kube-api-access-4n4zx\") on node \"ip-10-0-129-216.ec2.internal\" DevicePath \"\""
Jul 23 17:35:38.555162 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:35:38.555133    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fdf9d680-1864-4545-ac50-7b260943faa6-maas-api-tls\") on node \"ip-10-0-129-216.ec2.internal\" DevicePath \"\""
Jul 23 17:35:38.555162 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:35:38.555147    2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fdf9d680-1864-4545-ac50-7b260943faa6-openshift-service-ca\") on node \"ip-10-0-129-216.ec2.internal\" DevicePath \"\""
Jul 23 17:35:38.570854 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:35:38.570832    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-e6ac8a29-564695df4d-jgfwq"]
Jul 23 17:35:38.576424 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:35:38.576394    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-e6ac8a29-564695df4d-jgfwq"]
Jul 23 17:35:38.766811 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:35:38.766777    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fdf9d680-1864-4545-ac50-7b260943faa6" path="/var/lib/kubelet/pods/fdf9d680-1864-4545-ac50-7b260943faa6/volumes"
Jul 23 17:35:42.506462 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:35:42.506420    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e6ac8a29-db5wf"]
Jul 23 17:35:42.510233 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:35:42.510201    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:35:42.510670 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:35:42.510651    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e6ac8a29-db5wf"]
Jul 23 17:35:42.673280 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:35:42.673226    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:35:42.766312 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:35:42.766202    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9c14974c-4d7d-47e0-bdc6-a896d1d4918b" path="/var/lib/kubelet/pods/9c14974c-4d7d-47e0-bdc6-a896d1d4918b/volumes"
Jul 23 17:35:42.913440 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:35:42.913406    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:35:43.424769 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:35:43.424734    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:35:43.526865 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:35:43.526834    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:35:43.626235 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:35:43.626198    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:35:45.151296 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:35:45.151239    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:35:51.390769 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:35:51.390729    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:35:51.491822 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:35:51.491777    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:35:51.593488 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:35:51.593457    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:35:51.690605 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:35:51.690570    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:35:52.108791 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:35:52.108706    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:36:18.126222 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:36:18.126189    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:36:24.653911 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:36:24.653866    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:36:24.766897 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:36:24.766864    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:36:24.854546 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:36:24.854512    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:36:40.794550 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:36:40.794460    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:36:47.136581 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:36:47.136536    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:36:49.743937 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:36:49.743903    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:36:57.933043 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:36:57.933003    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:36:58.822424 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:36:58.822391    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:37:06.638585 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:37:06.638548    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:37:06.757005 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:37:06.756971    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:37:07.370302 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:37:07.370236    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:37:07.470891 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:37:07.470862    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:37:07.600799 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:37:07.600772    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:37:09.361524 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:37:09.361488    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:37:15.671626 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:37:15.671572    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:37:15.772347 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:37:15.772316    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:37:16.412130 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:37:16.412087    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:37:24.762042 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:37:24.762004    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:37:24.860037 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:37:24.860000    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:37:25.024885 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:37:25.024803    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:37:25.383370 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:37:25.383282    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:37:25.788723 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:37:25.788688    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:37:28.863614 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:37:28.863578    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:37:28.980354 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:37:28.980312    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:37:29.055719 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:37:29.055690    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c99bbc71-577dd4b6dc-mxqwq"]
Jul 23 17:37:29.056092 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:37:29.056081    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fdf9d680-1864-4545-ac50-7b260943faa6" containerName="maas-api"
Jul 23 17:37:29.056137 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:37:29.056094    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="fdf9d680-1864-4545-ac50-7b260943faa6" containerName="maas-api"
Jul 23 17:37:29.056137 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:37:29.056116    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c14974c-4d7d-47e0-bdc6-a896d1d4918b" containerName="revoke-keys"
Jul 23 17:37:29.056137 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:37:29.056121    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c14974c-4d7d-47e0-bdc6-a896d1d4918b" containerName="revoke-keys"
Jul 23 17:37:29.056230 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:37:29.056193    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="fdf9d680-1864-4545-ac50-7b260943faa6" containerName="maas-api"
Jul 23 17:37:29.056230 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:37:29.056201    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="9c14974c-4d7d-47e0-bdc6-a896d1d4918b" containerName="revoke-keys"
Jul 23 17:37:29.058419 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:37:29.058392    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c99bbc71-577dd4b6dc-mxqwq"
Jul 23 17:37:29.061248 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:37:29.061223    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-c99bbc71-serving-cert\""
Jul 23 17:37:29.099554 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:37:29.099528    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c99bbc71-577dd4b6dc-mxqwq"]
Jul 23 17:37:29.137223 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:37:29.137144    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kx9qq\" (UniqueName: \"kubernetes.io/projected/d4e1f3cb-ba06-44a4-81da-440f3d0b5c2e-kube-api-access-kx9qq\") pod \"maas-api-e2e-mt-c99bbc71-577dd4b6dc-mxqwq\" (UID: \"d4e1f3cb-ba06-44a4-81da-440f3d0b5c2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c99bbc71-577dd4b6dc-mxqwq"
Jul 23 17:37:29.137223 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:37:29.137205    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d4e1f3cb-ba06-44a4-81da-440f3d0b5c2e-maas-api-tls\") pod \"maas-api-e2e-mt-c99bbc71-577dd4b6dc-mxqwq\" (UID: \"d4e1f3cb-ba06-44a4-81da-440f3d0b5c2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c99bbc71-577dd4b6dc-mxqwq"
Jul 23 17:37:29.137407 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:37:29.137237    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4e1f3cb-ba06-44a4-81da-440f3d0b5c2e-openshift-service-ca\") pod \"maas-api-e2e-mt-c99bbc71-577dd4b6dc-mxqwq\" (UID: \"d4e1f3cb-ba06-44a4-81da-440f3d0b5c2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c99bbc71-577dd4b6dc-mxqwq"
Jul 23 17:37:29.197661 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:37:29.197630    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:37:29.237874 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:37:29.237843    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kx9qq\" (UniqueName: \"kubernetes.io/projected/d4e1f3cb-ba06-44a4-81da-440f3d0b5c2e-kube-api-access-kx9qq\") pod \"maas-api-e2e-mt-c99bbc71-577dd4b6dc-mxqwq\" (UID: \"d4e1f3cb-ba06-44a4-81da-440f3d0b5c2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c99bbc71-577dd4b6dc-mxqwq"
Jul 23 17:37:29.238023 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:37:29.237895    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d4e1f3cb-ba06-44a4-81da-440f3d0b5c2e-maas-api-tls\") pod \"maas-api-e2e-mt-c99bbc71-577dd4b6dc-mxqwq\" (UID: \"d4e1f3cb-ba06-44a4-81da-440f3d0b5c2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c99bbc71-577dd4b6dc-mxqwq"
Jul 23 17:37:29.238023 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:37:29.237916    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4e1f3cb-ba06-44a4-81da-440f3d0b5c2e-openshift-service-ca\") pod \"maas-api-e2e-mt-c99bbc71-577dd4b6dc-mxqwq\" (UID: \"d4e1f3cb-ba06-44a4-81da-440f3d0b5c2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c99bbc71-577dd4b6dc-mxqwq"
Jul 23 17:37:29.238023 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:37:29.237997    2577 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-c99bbc71-serving-cert: secret "maas-api-e2e-mt-c99bbc71-serving-cert" not found
Jul 23 17:37:29.238128 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:37:29.238074    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d4e1f3cb-ba06-44a4-81da-440f3d0b5c2e-maas-api-tls podName:d4e1f3cb-ba06-44a4-81da-440f3d0b5c2e nodeName:}" failed. No retries permitted until 2026-07-23 17:37:29.738054532 +0000 UTC m=+1945.508340042 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d4e1f3cb-ba06-44a4-81da-440f3d0b5c2e-maas-api-tls") pod "maas-api-e2e-mt-c99bbc71-577dd4b6dc-mxqwq" (UID: "d4e1f3cb-ba06-44a4-81da-440f3d0b5c2e") : secret "maas-api-e2e-mt-c99bbc71-serving-cert" not found
Jul 23 17:37:29.238532 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:37:29.238515    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4e1f3cb-ba06-44a4-81da-440f3d0b5c2e-openshift-service-ca\") pod \"maas-api-e2e-mt-c99bbc71-577dd4b6dc-mxqwq\" (UID: \"d4e1f3cb-ba06-44a4-81da-440f3d0b5c2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c99bbc71-577dd4b6dc-mxqwq"
Jul 23 17:37:29.252156 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:37:29.252126    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kx9qq\" (UniqueName: \"kubernetes.io/projected/d4e1f3cb-ba06-44a4-81da-440f3d0b5c2e-kube-api-access-kx9qq\") pod \"maas-api-e2e-mt-c99bbc71-577dd4b6dc-mxqwq\" (UID: \"d4e1f3cb-ba06-44a4-81da-440f3d0b5c2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c99bbc71-577dd4b6dc-mxqwq"
Jul 23 17:37:29.404352 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:37:29.404273    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:37:29.687450 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:37:29.687359    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:37:29.743143 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:37:29.743111    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d4e1f3cb-ba06-44a4-81da-440f3d0b5c2e-maas-api-tls\") pod \"maas-api-e2e-mt-c99bbc71-577dd4b6dc-mxqwq\" (UID: \"d4e1f3cb-ba06-44a4-81da-440f3d0b5c2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c99bbc71-577dd4b6dc-mxqwq"
Jul 23 17:37:29.745534 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:37:29.745506    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d4e1f3cb-ba06-44a4-81da-440f3d0b5c2e-maas-api-tls\") pod \"maas-api-e2e-mt-c99bbc71-577dd4b6dc-mxqwq\" (UID: \"d4e1f3cb-ba06-44a4-81da-440f3d0b5c2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c99bbc71-577dd4b6dc-mxqwq"
Jul 23 17:37:29.802244 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:37:29.802218    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:37:29.968855 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:37:29.968820    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c99bbc71-577dd4b6dc-mxqwq"
Jul 23 17:37:30.111148 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:37:30.111123    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c99bbc71-577dd4b6dc-mxqwq"]
Jul 23 17:37:30.112741 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:37:30.112710    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4e1f3cb_ba06_44a4_81da_440f3d0b5c2e.slice/crio-d58c630341513ee90d9da4df86dda97102ccf046f7261d2e0a5cc3880461ae50 WatchSource:0}: Error finding container d58c630341513ee90d9da4df86dda97102ccf046f7261d2e0a5cc3880461ae50: Status 404 returned error can't find the container with id d58c630341513ee90d9da4df86dda97102ccf046f7261d2e0a5cc3880461ae50
Jul 23 17:37:30.167676 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:37:30.167645    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c99bbc71-577dd4b6dc-mxqwq" event={"ID":"d4e1f3cb-ba06-44a4-81da-440f3d0b5c2e","Type":"ContainerStarted","Data":"d58c630341513ee90d9da4df86dda97102ccf046f7261d2e0a5cc3880461ae50"}
Jul 23 17:37:31.174919 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:37:31.174881    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c99bbc71-577dd4b6dc-mxqwq" event={"ID":"d4e1f3cb-ba06-44a4-81da-440f3d0b5c2e","Type":"ContainerStarted","Data":"87542bccf2b7dd58493bd32a39edf44e18e35217296e85d4a063b37985ee5ee8"}
Jul 23 17:37:31.175346 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:37:31.174952    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c99bbc71-577dd4b6dc-mxqwq"
Jul 23 17:37:31.195295 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:37:31.195228    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c99bbc71-577dd4b6dc-mxqwq" podStartSLOduration=1.6843722140000001 podStartE2EDuration="2.195213s" totalImagesPullingTime="510.840786ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 17:37:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 17:37:31.193996123 +0000 UTC m=+1946.964281666" watchObservedRunningTime="2026-07-23 17:37:31.195213 +0000 UTC m=+1946.965498519"
Jul 23 17:37:34.072464 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:37:34.072435    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:37:34.167926 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:37:34.167886    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:37:34.609323 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:37:34.609291    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:37:37.186213 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:37:37.186183    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c99bbc71-577dd4b6dc-mxqwq"
Jul 23 17:37:40.742438 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:37:40.742404    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:37:40.809278 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:37:40.809230    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:37:41.109395 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:37:41.109319    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c99bbc71-kfkz7"]
Jul 23 17:37:41.111991 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:37:41.111975    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c99bbc71-kfkz7"
Jul 23 17:37:41.114777 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:37:41.114754    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7wfmw\""
Jul 23 17:37:41.127898 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:37:41.127874    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c99bbc71-kfkz7"]
Jul 23 17:37:41.141666 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:37:41.141629    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a758822-81d1-44e9-8119-f34d22b6a940-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-c99bbc71-kfkz7\" (UID: \"3a758822-81d1-44e9-8119-f34d22b6a940\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c99bbc71-kfkz7"
Jul 23 17:37:41.242524 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:37:41.242497    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a758822-81d1-44e9-8119-f34d22b6a940-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-c99bbc71-kfkz7\" (UID: \"3a758822-81d1-44e9-8119-f34d22b6a940\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c99bbc71-kfkz7"
Jul 23 17:37:41.243101 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:37:41.243082    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a758822-81d1-44e9-8119-f34d22b6a940-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-c99bbc71-kfkz7\" (UID: \"3a758822-81d1-44e9-8119-f34d22b6a940\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c99bbc71-kfkz7"
Jul 23 17:37:41.422571 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:37:41.422486    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c99bbc71-kfkz7"
Jul 23 17:37:41.560416 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:37:41.560391    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c99bbc71-kfkz7"]
Jul 23 17:37:41.562871 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:37:41.562844    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a758822_81d1_44e9_8119_f34d22b6a940.slice/crio-6bc7aedc5b1347e20c23a1349ffe44d49ea6f4ae4b20c88d72eb6420860b5272 WatchSource:0}: Error finding container 6bc7aedc5b1347e20c23a1349ffe44d49ea6f4ae4b20c88d72eb6420860b5272: Status 404 returned error can't find the container with id 6bc7aedc5b1347e20c23a1349ffe44d49ea6f4ae4b20c88d72eb6420860b5272
Jul 23 17:37:42.238773 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:37:42.238740    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c99bbc71-kfkz7" event={"ID":"3a758822-81d1-44e9-8119-f34d22b6a940","Type":"ContainerStarted","Data":"ccafc5abd6e601a13657e4f07b7ac3d71ceef30aa2953870ea833e827ab07901"}
Jul 23 17:37:42.239123 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:37:42.238801    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c99bbc71-kfkz7" event={"ID":"3a758822-81d1-44e9-8119-f34d22b6a940","Type":"ContainerStarted","Data":"6bc7aedc5b1347e20c23a1349ffe44d49ea6f4ae4b20c88d72eb6420860b5272"}
Jul 23 17:37:42.261521 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:37:42.261479    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c99bbc71-kfkz7" podStartSLOduration=1.26146279 podStartE2EDuration="1.26146279s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 17:37:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 17:37:42.261015514 +0000 UTC m=+1958.031301033" watchObservedRunningTime="2026-07-23 17:37:42.26146279 +0000 UTC m=+1958.031748318"
Jul 23 17:37:43.245593 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:37:43.245561    2577 generic.go:356] "Generic (PLEG): container finished" podID="3a758822-81d1-44e9-8119-f34d22b6a940" containerID="ccafc5abd6e601a13657e4f07b7ac3d71ceef30aa2953870ea833e827ab07901" exitCode=0
Jul 23 17:37:43.245966 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:37:43.245649    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c99bbc71-kfkz7" event={"ID":"3a758822-81d1-44e9-8119-f34d22b6a940","Type":"ContainerDied","Data":"ccafc5abd6e601a13657e4f07b7ac3d71ceef30aa2953870ea833e827ab07901"}
Jul 23 17:37:44.386652 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:37:44.386620    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c99bbc71-kfkz7"
Jul 23 17:37:44.476508 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:37:44.476479    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a758822-81d1-44e9-8119-f34d22b6a940-maas-api-service-ca\") pod \"3a758822-81d1-44e9-8119-f34d22b6a940\" (UID: \"3a758822-81d1-44e9-8119-f34d22b6a940\") "
Jul 23 17:37:44.476804 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:37:44.476782    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a758822-81d1-44e9-8119-f34d22b6a940-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "3a758822-81d1-44e9-8119-f34d22b6a940" (UID: "3a758822-81d1-44e9-8119-f34d22b6a940"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 17:37:44.577923 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:37:44.577845    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a758822-81d1-44e9-8119-f34d22b6a940-maas-api-service-ca\") on node \"ip-10-0-129-216.ec2.internal\" DevicePath \"\""
Jul 23 17:37:45.258387 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:37:45.258351    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c99bbc71-kfkz7"
Jul 23 17:37:45.258537 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:37:45.258386    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c99bbc71-kfkz7" event={"ID":"3a758822-81d1-44e9-8119-f34d22b6a940","Type":"ContainerDied","Data":"6bc7aedc5b1347e20c23a1349ffe44d49ea6f4ae4b20c88d72eb6420860b5272"}
Jul 23 17:37:45.258537 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:37:45.258462    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6bc7aedc5b1347e20c23a1349ffe44d49ea6f4ae4b20c88d72eb6420860b5272"
Jul 23 17:37:51.672720 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:37:51.672688    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c99bbc71-577dd4b6dc-mxqwq"]
Jul 23 17:37:51.673302 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:37:51.672918    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c99bbc71-577dd4b6dc-mxqwq" podUID="d4e1f3cb-ba06-44a4-81da-440f3d0b5c2e" containerName="maas-api" containerID="cri-o://87542bccf2b7dd58493bd32a39edf44e18e35217296e85d4a063b37985ee5ee8" gracePeriod=30
Jul 23 17:37:51.699507 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:37:51.699486    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:37:51.920220 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:37:51.920191    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c99bbc71-577dd4b6dc-mxqwq"
Jul 23 17:37:52.045791 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:37:52.045752    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kx9qq\" (UniqueName: \"kubernetes.io/projected/d4e1f3cb-ba06-44a4-81da-440f3d0b5c2e-kube-api-access-kx9qq\") pod \"d4e1f3cb-ba06-44a4-81da-440f3d0b5c2e\" (UID: \"d4e1f3cb-ba06-44a4-81da-440f3d0b5c2e\") "
Jul 23 17:37:52.046004 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:37:52.045825    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4e1f3cb-ba06-44a4-81da-440f3d0b5c2e-openshift-service-ca\") pod \"d4e1f3cb-ba06-44a4-81da-440f3d0b5c2e\" (UID: \"d4e1f3cb-ba06-44a4-81da-440f3d0b5c2e\") "
Jul 23 17:37:52.046004 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:37:52.045888    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d4e1f3cb-ba06-44a4-81da-440f3d0b5c2e-maas-api-tls\") pod \"d4e1f3cb-ba06-44a4-81da-440f3d0b5c2e\" (UID: \"d4e1f3cb-ba06-44a4-81da-440f3d0b5c2e\") "
Jul 23 17:37:52.046182 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:37:52.046153    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4e1f3cb-ba06-44a4-81da-440f3d0b5c2e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "d4e1f3cb-ba06-44a4-81da-440f3d0b5c2e" (UID: "d4e1f3cb-ba06-44a4-81da-440f3d0b5c2e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 17:37:52.047987 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:37:52.047963    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4e1f3cb-ba06-44a4-81da-440f3d0b5c2e-kube-api-access-kx9qq" (OuterVolumeSpecName: "kube-api-access-kx9qq") pod "d4e1f3cb-ba06-44a4-81da-440f3d0b5c2e" (UID: "d4e1f3cb-ba06-44a4-81da-440f3d0b5c2e"). InnerVolumeSpecName "kube-api-access-kx9qq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 17:37:52.048091 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:37:52.048059    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d4e1f3cb-ba06-44a4-81da-440f3d0b5c2e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "d4e1f3cb-ba06-44a4-81da-440f3d0b5c2e" (UID: "d4e1f3cb-ba06-44a4-81da-440f3d0b5c2e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 17:37:52.147528 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:37:52.147483    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d4e1f3cb-ba06-44a4-81da-440f3d0b5c2e-maas-api-tls\") on node \"ip-10-0-129-216.ec2.internal\" DevicePath \"\""
Jul 23 17:37:52.147528 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:37:52.147523    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kx9qq\" (UniqueName: \"kubernetes.io/projected/d4e1f3cb-ba06-44a4-81da-440f3d0b5c2e-kube-api-access-kx9qq\") on node \"ip-10-0-129-216.ec2.internal\" DevicePath \"\""
Jul 23 17:37:52.147704 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:37:52.147540    2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4e1f3cb-ba06-44a4-81da-440f3d0b5c2e-openshift-service-ca\") on node \"ip-10-0-129-216.ec2.internal\" DevicePath \"\""
Jul 23 17:37:52.297498 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:37:52.297407    2577 generic.go:356] "Generic (PLEG): container finished" podID="d4e1f3cb-ba06-44a4-81da-440f3d0b5c2e" containerID="87542bccf2b7dd58493bd32a39edf44e18e35217296e85d4a063b37985ee5ee8" exitCode=0
Jul 23 17:37:52.297498 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:37:52.297447    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c99bbc71-577dd4b6dc-mxqwq"
Jul 23 17:37:52.297696 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:37:52.297495    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c99bbc71-577dd4b6dc-mxqwq" event={"ID":"d4e1f3cb-ba06-44a4-81da-440f3d0b5c2e","Type":"ContainerDied","Data":"87542bccf2b7dd58493bd32a39edf44e18e35217296e85d4a063b37985ee5ee8"}
Jul 23 17:37:52.297696 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:37:52.297544    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c99bbc71-577dd4b6dc-mxqwq" event={"ID":"d4e1f3cb-ba06-44a4-81da-440f3d0b5c2e","Type":"ContainerDied","Data":"d58c630341513ee90d9da4df86dda97102ccf046f7261d2e0a5cc3880461ae50"}
Jul 23 17:37:52.297696 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:37:52.297572    2577 scope.go:117] "RemoveContainer" containerID="87542bccf2b7dd58493bd32a39edf44e18e35217296e85d4a063b37985ee5ee8"
Jul 23 17:37:52.307983 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:37:52.307968    2577 scope.go:117] "RemoveContainer" containerID="87542bccf2b7dd58493bd32a39edf44e18e35217296e85d4a063b37985ee5ee8"
Jul 23 17:37:52.308228 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:37:52.308209    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87542bccf2b7dd58493bd32a39edf44e18e35217296e85d4a063b37985ee5ee8\": container with ID starting with 87542bccf2b7dd58493bd32a39edf44e18e35217296e85d4a063b37985ee5ee8 not found: ID does not exist" containerID="87542bccf2b7dd58493bd32a39edf44e18e35217296e85d4a063b37985ee5ee8"
Jul 23 17:37:52.308302 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:37:52.308241    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87542bccf2b7dd58493bd32a39edf44e18e35217296e85d4a063b37985ee5ee8"} err="failed to get container status \"87542bccf2b7dd58493bd32a39edf44e18e35217296e85d4a063b37985ee5ee8\": rpc error: code = NotFound desc = could not find container \"87542bccf2b7dd58493bd32a39edf44e18e35217296e85d4a063b37985ee5ee8\": container with ID starting with 87542bccf2b7dd58493bd32a39edf44e18e35217296e85d4a063b37985ee5ee8 not found: ID does not exist"
Jul 23 17:37:52.319632 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:37:52.319603    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c99bbc71-577dd4b6dc-mxqwq"]
Jul 23 17:37:52.325214 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:37:52.325193    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c99bbc71-577dd4b6dc-mxqwq"]
Jul 23 17:37:52.765981 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:37:52.765936    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d4e1f3cb-ba06-44a4-81da-440f3d0b5c2e" path="/var/lib/kubelet/pods/d4e1f3cb-ba06-44a4-81da-440f3d0b5c2e/volumes"
Jul 23 17:37:56.228922 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:37:56.228884    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c99bbc71-kfkz7"]
Jul 23 17:37:56.234560 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:37:56.234524    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c99bbc71-kfkz7"]
Jul 23 17:37:56.631637 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:37:56.631494    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:37:56.738169 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:37:56.738131    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:37:56.765986 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:37:56.765956    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3a758822-81d1-44e9-8119-f34d22b6a940" path="/var/lib/kubelet/pods/3a758822-81d1-44e9-8119-f34d22b6a940/volumes"
Jul 23 17:38:04.358203 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:38:04.358157    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:38:04.465405 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:38:04.465368    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:38:04.685650 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:38:04.685565    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:38:04.972841 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:38:04.972811    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:38:05.113096 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:38:05.113025    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:38:05.214779 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:38:05.214735    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:38:05.428483 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:38:05.428354    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:38:12.653364 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:38:12.653327    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:38:12.778690 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:38:12.778648    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:38:12.877405 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:38:12.877361    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:38:13.013358 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:38:13.013324    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:38:21.406386 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:38:21.406340    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:38:21.511679 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:38:21.511638    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:38:30.656269 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:38:30.656222    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:38:30.775645 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:38:30.775609    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:38:31.284837 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:38:31.284801    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:38:33.215013 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:38:33.214967    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:38:33.314051 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:38:33.314019    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7a6850d7-f9d7fcd65-5jd55"]
Jul 23 17:38:33.314496 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:38:33.314481    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a758822-81d1-44e9-8119-f34d22b6a940" containerName="revoke-keys"
Jul 23 17:38:33.314496 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:38:33.314498    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a758822-81d1-44e9-8119-f34d22b6a940" containerName="revoke-keys"
Jul 23 17:38:33.314590 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:38:33.314511    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d4e1f3cb-ba06-44a4-81da-440f3d0b5c2e" containerName="maas-api"
Jul 23 17:38:33.314590 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:38:33.314517    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="d4e1f3cb-ba06-44a4-81da-440f3d0b5c2e" containerName="maas-api"
Jul 23 17:38:33.314590 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:38:33.314582    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="d4e1f3cb-ba06-44a4-81da-440f3d0b5c2e" containerName="maas-api"
Jul 23 17:38:33.314590 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:38:33.314591    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="3a758822-81d1-44e9-8119-f34d22b6a940" containerName="revoke-keys"
Jul 23 17:38:33.318017 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:38:33.317994    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7a6850d7-f9d7fcd65-5jd55"
Jul 23 17:38:33.322603 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:38:33.322582    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-7a6850d7-serving-cert\""
Jul 23 17:38:33.324087 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:38:33.324063    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:38:33.334249 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:38:33.334221    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7a6850d7-f9d7fcd65-5jd55"]
Jul 23 17:38:33.410709 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:38:33.410679    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0f8f138f-b51a-4c62-9e95-47e578e25e1d-maas-api-tls\") pod \"maas-api-e2e-mt-7a6850d7-f9d7fcd65-5jd55\" (UID: \"0f8f138f-b51a-4c62-9e95-47e578e25e1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7a6850d7-f9d7fcd65-5jd55"
Jul 23 17:38:33.410903 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:38:33.410730    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f8f138f-b51a-4c62-9e95-47e578e25e1d-openshift-service-ca\") pod \"maas-api-e2e-mt-7a6850d7-f9d7fcd65-5jd55\" (UID: \"0f8f138f-b51a-4c62-9e95-47e578e25e1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7a6850d7-f9d7fcd65-5jd55"
Jul 23 17:38:33.410903 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:38:33.410759    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hqd6s\" (UniqueName: \"kubernetes.io/projected/0f8f138f-b51a-4c62-9e95-47e578e25e1d-kube-api-access-hqd6s\") pod \"maas-api-e2e-mt-7a6850d7-f9d7fcd65-5jd55\" (UID: \"0f8f138f-b51a-4c62-9e95-47e578e25e1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7a6850d7-f9d7fcd65-5jd55"
Jul 23 17:38:33.432100 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:38:33.432071    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:38:33.511535 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:38:33.511448    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f8f138f-b51a-4c62-9e95-47e578e25e1d-openshift-service-ca\") pod \"maas-api-e2e-mt-7a6850d7-f9d7fcd65-5jd55\" (UID: \"0f8f138f-b51a-4c62-9e95-47e578e25e1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7a6850d7-f9d7fcd65-5jd55"
Jul 23 17:38:33.511535 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:38:33.511489    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqd6s\" (UniqueName: \"kubernetes.io/projected/0f8f138f-b51a-4c62-9e95-47e578e25e1d-kube-api-access-hqd6s\") pod \"maas-api-e2e-mt-7a6850d7-f9d7fcd65-5jd55\" (UID: \"0f8f138f-b51a-4c62-9e95-47e578e25e1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7a6850d7-f9d7fcd65-5jd55"
Jul 23 17:38:33.511754 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:38:33.511570    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0f8f138f-b51a-4c62-9e95-47e578e25e1d-maas-api-tls\") pod \"maas-api-e2e-mt-7a6850d7-f9d7fcd65-5jd55\" (UID: \"0f8f138f-b51a-4c62-9e95-47e578e25e1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7a6850d7-f9d7fcd65-5jd55"
Jul 23 17:38:33.512082 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:38:33.512058    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f8f138f-b51a-4c62-9e95-47e578e25e1d-openshift-service-ca\") pod \"maas-api-e2e-mt-7a6850d7-f9d7fcd65-5jd55\" (UID: \"0f8f138f-b51a-4c62-9e95-47e578e25e1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7a6850d7-f9d7fcd65-5jd55"
Jul 23 17:38:33.514069 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:38:33.514028    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0f8f138f-b51a-4c62-9e95-47e578e25e1d-maas-api-tls\") pod \"maas-api-e2e-mt-7a6850d7-f9d7fcd65-5jd55\" (UID: \"0f8f138f-b51a-4c62-9e95-47e578e25e1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7a6850d7-f9d7fcd65-5jd55"
Jul 23 17:38:33.526976 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:38:33.526947    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hqd6s\" (UniqueName: \"kubernetes.io/projected/0f8f138f-b51a-4c62-9e95-47e578e25e1d-kube-api-access-hqd6s\") pod \"maas-api-e2e-mt-7a6850d7-f9d7fcd65-5jd55\" (UID: \"0f8f138f-b51a-4c62-9e95-47e578e25e1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7a6850d7-f9d7fcd65-5jd55"
Jul 23 17:38:33.604465 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:38:33.604430    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:38:33.628366 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:38:33.628340    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7a6850d7-f9d7fcd65-5jd55"
Jul 23 17:38:33.786915 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:38:33.786884    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7a6850d7-f9d7fcd65-5jd55"]
Jul 23 17:38:33.788358 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:38:33.788328    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f8f138f_b51a_4c62_9e95_47e578e25e1d.slice/crio-81967e5682556dd9f573c7375c7e320751db02832b3081eeb3575d4af6ec4117 WatchSource:0}: Error finding container 81967e5682556dd9f573c7375c7e320751db02832b3081eeb3575d4af6ec4117: Status 404 returned error can't find the container with id 81967e5682556dd9f573c7375c7e320751db02832b3081eeb3575d4af6ec4117
Jul 23 17:38:33.789726 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:38:33.789707    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 23 17:38:34.121502 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:38:34.121418    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:38:34.231144 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:38:34.231106    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:38:34.531502 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:38:34.531464    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7a6850d7-f9d7fcd65-5jd55" event={"ID":"0f8f138f-b51a-4c62-9e95-47e578e25e1d","Type":"ContainerStarted","Data":"8a9685677100218be5717e689b07a3cf7d2c722752e10d04cf4ed0fe1ba99e57"}
Jul 23 17:38:34.531852 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:38:34.531831    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7a6850d7-f9d7fcd65-5jd55"
Jul 23 17:38:34.532225 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:38:34.532197    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7a6850d7-f9d7fcd65-5jd55" event={"ID":"0f8f138f-b51a-4c62-9e95-47e578e25e1d","Type":"ContainerStarted","Data":"81967e5682556dd9f573c7375c7e320751db02832b3081eeb3575d4af6ec4117"}
Jul 23 17:38:34.585166 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:38:34.585115    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7a6850d7-f9d7fcd65-5jd55" podStartSLOduration=1.135416967 podStartE2EDuration="1.585097838s" totalImagesPullingTime="449.680871ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 17:38:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 17:38:34.58348518 +0000 UTC m=+2010.353770700" watchObservedRunningTime="2026-07-23 17:38:34.585097838 +0000 UTC m=+2010.355383357"
Jul 23 17:38:37.844861 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:38:37.844822    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:38:41.548995 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:38:41.548960    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7a6850d7-f9d7fcd65-5jd55"
Jul 23 17:38:46.408019 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:38:46.407984    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-7a6850d7/e2e-lifecycle-model-7a6850d7-kserve-6b965cd5d7-9g9x8"]
Jul 23 17:38:46.411583 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:38:46.411560    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-7a6850d7/e2e-lifecycle-model-7a6850d7-kserve-6b965cd5d7-9g9x8"
Jul 23 17:38:46.415396 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:38:46.415371    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-7a6850d7\"/\"openshift-service-ca.crt\""
Jul 23 17:38:46.415532 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:38:46.415373    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-7a6850d7\"/\"default-dockercfg-5nnsz\""
Jul 23 17:38:46.415532 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:38:46.415424    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-7a6850d7\"/\"kube-root-ca.crt\""
Jul 23 17:38:46.415532 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:38:46.415371    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-7a6850d7\"/\"e2e-lifecycle-model-7a6850d7-kserve-self-signed-certs\""
Jul 23 17:38:46.424996 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:38:46.424970    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-7a6850d7/e2e-lifecycle-model-7a6850d7-kserve-6b965cd5d7-9g9x8"]
Jul 23 17:38:46.448379 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:38:46.448352    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:38:46.536005 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:38:46.535968    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b07ed6e-4258-4bce-adb0-4285527c875f-home\") pod \"e2e-lifecycle-model-7a6850d7-kserve-6b965cd5d7-9g9x8\" (UID: \"5b07ed6e-4258-4bce-adb0-4285527c875f\") " pod="ai-tenant-e2e-mt-7a6850d7/e2e-lifecycle-model-7a6850d7-kserve-6b965cd5d7-9g9x8"
Jul 23 17:38:46.536186 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:38:46.536036    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b07ed6e-4258-4bce-adb0-4285527c875f-model-cache\") pod \"e2e-lifecycle-model-7a6850d7-kserve-6b965cd5d7-9g9x8\" (UID: \"5b07ed6e-4258-4bce-adb0-4285527c875f\") " pod="ai-tenant-e2e-mt-7a6850d7/e2e-lifecycle-model-7a6850d7-kserve-6b965cd5d7-9g9x8"
Jul 23 17:38:46.536186 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:38:46.536069    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wt4fq\" (UniqueName: \"kubernetes.io/projected/5b07ed6e-4258-4bce-adb0-4285527c875f-kube-api-access-wt4fq\") pod \"e2e-lifecycle-model-7a6850d7-kserve-6b965cd5d7-9g9x8\" (UID: \"5b07ed6e-4258-4bce-adb0-4285527c875f\") " pod="ai-tenant-e2e-mt-7a6850d7/e2e-lifecycle-model-7a6850d7-kserve-6b965cd5d7-9g9x8"
Jul 23 17:38:46.536186 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:38:46.536108    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b07ed6e-4258-4bce-adb0-4285527c875f-tmp-dir\") pod \"e2e-lifecycle-model-7a6850d7-kserve-6b965cd5d7-9g9x8\" (UID: \"5b07ed6e-4258-4bce-adb0-4285527c875f\") " pod="ai-tenant-e2e-mt-7a6850d7/e2e-lifecycle-model-7a6850d7-kserve-6b965cd5d7-9g9x8"
Jul 23 17:38:46.536186 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:38:46.536164    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b07ed6e-4258-4bce-adb0-4285527c875f-dshm\") pod \"e2e-lifecycle-model-7a6850d7-kserve-6b965cd5d7-9g9x8\" (UID: \"5b07ed6e-4258-4bce-adb0-4285527c875f\") " pod="ai-tenant-e2e-mt-7a6850d7/e2e-lifecycle-model-7a6850d7-kserve-6b965cd5d7-9g9x8"
Jul 23 17:38:46.536452 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:38:46.536230    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b07ed6e-4258-4bce-adb0-4285527c875f-tls-certs\") pod \"e2e-lifecycle-model-7a6850d7-kserve-6b965cd5d7-9g9x8\" (UID: \"5b07ed6e-4258-4bce-adb0-4285527c875f\") " pod="ai-tenant-e2e-mt-7a6850d7/e2e-lifecycle-model-7a6850d7-kserve-6b965cd5d7-9g9x8"
Jul 23 17:38:46.637444 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:38:46.637409    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b07ed6e-4258-4bce-adb0-4285527c875f-tmp-dir\") pod \"e2e-lifecycle-model-7a6850d7-kserve-6b965cd5d7-9g9x8\" (UID: \"5b07ed6e-4258-4bce-adb0-4285527c875f\") " pod="ai-tenant-e2e-mt-7a6850d7/e2e-lifecycle-model-7a6850d7-kserve-6b965cd5d7-9g9x8"
Jul 23 17:38:46.637618 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:38:46.637454    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b07ed6e-4258-4bce-adb0-4285527c875f-dshm\") pod \"e2e-lifecycle-model-7a6850d7-kserve-6b965cd5d7-9g9x8\" (UID: \"5b07ed6e-4258-4bce-adb0-4285527c875f\") " pod="ai-tenant-e2e-mt-7a6850d7/e2e-lifecycle-model-7a6850d7-kserve-6b965cd5d7-9g9x8"
Jul 23 17:38:46.637618 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:38:46.637499    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b07ed6e-4258-4bce-adb0-4285527c875f-tls-certs\") pod \"e2e-lifecycle-model-7a6850d7-kserve-6b965cd5d7-9g9x8\" (UID: \"5b07ed6e-4258-4bce-adb0-4285527c875f\") " pod="ai-tenant-e2e-mt-7a6850d7/e2e-lifecycle-model-7a6850d7-kserve-6b965cd5d7-9g9x8"
Jul 23 17:38:46.637618 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:38:46.637540    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b07ed6e-4258-4bce-adb0-4285527c875f-home\") pod \"e2e-lifecycle-model-7a6850d7-kserve-6b965cd5d7-9g9x8\" (UID: \"5b07ed6e-4258-4bce-adb0-4285527c875f\") " pod="ai-tenant-e2e-mt-7a6850d7/e2e-lifecycle-model-7a6850d7-kserve-6b965cd5d7-9g9x8"
Jul 23 17:38:46.637791 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:38:46.637701    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b07ed6e-4258-4bce-adb0-4285527c875f-model-cache\") pod \"e2e-lifecycle-model-7a6850d7-kserve-6b965cd5d7-9g9x8\" (UID: \"5b07ed6e-4258-4bce-adb0-4285527c875f\") " pod="ai-tenant-e2e-mt-7a6850d7/e2e-lifecycle-model-7a6850d7-kserve-6b965cd5d7-9g9x8"
Jul 23 17:38:46.637791 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:38:46.637750    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wt4fq\" (UniqueName: \"kubernetes.io/projected/5b07ed6e-4258-4bce-adb0-4285527c875f-kube-api-access-wt4fq\") pod \"e2e-lifecycle-model-7a6850d7-kserve-6b965cd5d7-9g9x8\" (UID: \"5b07ed6e-4258-4bce-adb0-4285527c875f\") " pod="ai-tenant-e2e-mt-7a6850d7/e2e-lifecycle-model-7a6850d7-kserve-6b965cd5d7-9g9x8"
Jul 23 17:38:46.637894 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:38:46.637865    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b07ed6e-4258-4bce-adb0-4285527c875f-tmp-dir\") pod \"e2e-lifecycle-model-7a6850d7-kserve-6b965cd5d7-9g9x8\" (UID: \"5b07ed6e-4258-4bce-adb0-4285527c875f\") " pod="ai-tenant-e2e-mt-7a6850d7/e2e-lifecycle-model-7a6850d7-kserve-6b965cd5d7-9g9x8"
Jul 23 17:38:46.637945 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:38:46.637896    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b07ed6e-4258-4bce-adb0-4285527c875f-home\") pod \"e2e-lifecycle-model-7a6850d7-kserve-6b965cd5d7-9g9x8\" (UID: \"5b07ed6e-4258-4bce-adb0-4285527c875f\") " pod="ai-tenant-e2e-mt-7a6850d7/e2e-lifecycle-model-7a6850d7-kserve-6b965cd5d7-9g9x8"
Jul 23 17:38:46.638067 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:38:46.638045    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b07ed6e-4258-4bce-adb0-4285527c875f-model-cache\") pod \"e2e-lifecycle-model-7a6850d7-kserve-6b965cd5d7-9g9x8\" (UID: \"5b07ed6e-4258-4bce-adb0-4285527c875f\") " pod="ai-tenant-e2e-mt-7a6850d7/e2e-lifecycle-model-7a6850d7-kserve-6b965cd5d7-9g9x8"
Jul 23 17:38:46.639886 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:38:46.639862    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b07ed6e-4258-4bce-adb0-4285527c875f-dshm\") pod \"e2e-lifecycle-model-7a6850d7-kserve-6b965cd5d7-9g9x8\" (UID: \"5b07ed6e-4258-4bce-adb0-4285527c875f\") " pod="ai-tenant-e2e-mt-7a6850d7/e2e-lifecycle-model-7a6850d7-kserve-6b965cd5d7-9g9x8"
Jul 23 17:38:46.640116 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:38:46.640098    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b07ed6e-4258-4bce-adb0-4285527c875f-tls-certs\") pod \"e2e-lifecycle-model-7a6850d7-kserve-6b965cd5d7-9g9x8\" (UID: \"5b07ed6e-4258-4bce-adb0-4285527c875f\") " pod="ai-tenant-e2e-mt-7a6850d7/e2e-lifecycle-model-7a6850d7-kserve-6b965cd5d7-9g9x8"
Jul 23 17:38:46.655451 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:38:46.655432    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wt4fq\" (UniqueName: \"kubernetes.io/projected/5b07ed6e-4258-4bce-adb0-4285527c875f-kube-api-access-wt4fq\") pod \"e2e-lifecycle-model-7a6850d7-kserve-6b965cd5d7-9g9x8\" (UID: \"5b07ed6e-4258-4bce-adb0-4285527c875f\") " pod="ai-tenant-e2e-mt-7a6850d7/e2e-lifecycle-model-7a6850d7-kserve-6b965cd5d7-9g9x8"
Jul 23 17:38:46.722390 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:38:46.722366    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-7a6850d7/e2e-lifecycle-model-7a6850d7-kserve-6b965cd5d7-9g9x8"
Jul 23 17:38:46.871082 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:38:46.871056    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-7a6850d7/e2e-lifecycle-model-7a6850d7-kserve-6b965cd5d7-9g9x8"]
Jul 23 17:38:46.871776 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:38:46.871752    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5b07ed6e_4258_4bce_adb0_4285527c875f.slice/crio-a989e3ee2cd9d3e9eddbbd047268489367ea709ce25ee14568f51d1bb7564a59 WatchSource:0}: Error finding container a989e3ee2cd9d3e9eddbbd047268489367ea709ce25ee14568f51d1bb7564a59: Status 404 returned error can't find the container with id a989e3ee2cd9d3e9eddbbd047268489367ea709ce25ee14568f51d1bb7564a59
Jul 23 17:38:47.613337 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:38:47.613298    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-7a6850d7/e2e-lifecycle-model-7a6850d7-kserve-6b965cd5d7-9g9x8" event={"ID":"5b07ed6e-4258-4bce-adb0-4285527c875f","Type":"ContainerStarted","Data":"a88a8b2c64d5f681bad8ea3a56853067e826047800ade08b8fdf1a189c065af4"}
Jul 23 17:38:47.613784 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:38:47.613346    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-7a6850d7/e2e-lifecycle-model-7a6850d7-kserve-6b965cd5d7-9g9x8" event={"ID":"5b07ed6e-4258-4bce-adb0-4285527c875f","Type":"ContainerStarted","Data":"a989e3ee2cd9d3e9eddbbd047268489367ea709ce25ee14568f51d1bb7564a59"}
Jul 23 17:38:47.650007 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:38:47.649949    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-7a6850d7/e2e-lifecycle-model-7a6850d7-kserve-6b965cd5d7-9g9x8" podStartSLOduration=1.649931746 podStartE2EDuration="1.649931746s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 17:38:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 17:38:47.647570427 +0000 UTC m=+2023.417855948" watchObservedRunningTime="2026-07-23 17:38:47.649931746 +0000 UTC m=+2023.420217265"
Jul 23 17:38:56.723378 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:38:56.723340    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-7a6850d7/e2e-lifecycle-model-7a6850d7-kserve-6b965cd5d7-9g9x8"
Jul 23 17:38:56.723760 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:38:56.723402    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-7a6850d7/e2e-lifecycle-model-7a6850d7-kserve-6b965cd5d7-9g9x8"
Jul 23 17:38:56.735995 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:38:56.735968    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-7a6850d7/e2e-lifecycle-model-7a6850d7-kserve-6b965cd5d7-9g9x8"
Jul 23 17:38:57.681426 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:38:57.681397    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-7a6850d7/e2e-lifecycle-model-7a6850d7-kserve-6b965cd5d7-9g9x8"
Jul 23 17:38:59.975791 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:38:59.975758    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:39:26.959304 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:39:26.959182    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7a6850d7-f9d7fcd65-5jd55"]
Jul 23 17:39:26.959884 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:39:26.959411    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7a6850d7-f9d7fcd65-5jd55" podUID="0f8f138f-b51a-4c62-9e95-47e578e25e1d" containerName="maas-api" containerID="cri-o://8a9685677100218be5717e689b07a3cf7d2c722752e10d04cf4ed0fe1ba99e57" gracePeriod=30
Jul 23 17:39:26.991462 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:39:26.991438    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:39:27.209203 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:39:27.209179    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7a6850d7-f9d7fcd65-5jd55"
Jul 23 17:39:27.300488 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:39:27.300411    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f8f138f-b51a-4c62-9e95-47e578e25e1d-openshift-service-ca\") pod \"0f8f138f-b51a-4c62-9e95-47e578e25e1d\" (UID: \"0f8f138f-b51a-4c62-9e95-47e578e25e1d\") "
Jul 23 17:39:27.300488 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:39:27.300473    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hqd6s\" (UniqueName: \"kubernetes.io/projected/0f8f138f-b51a-4c62-9e95-47e578e25e1d-kube-api-access-hqd6s\") pod \"0f8f138f-b51a-4c62-9e95-47e578e25e1d\" (UID: \"0f8f138f-b51a-4c62-9e95-47e578e25e1d\") "
Jul 23 17:39:27.300702 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:39:27.300531    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0f8f138f-b51a-4c62-9e95-47e578e25e1d-maas-api-tls\") pod \"0f8f138f-b51a-4c62-9e95-47e578e25e1d\" (UID: \"0f8f138f-b51a-4c62-9e95-47e578e25e1d\") "
Jul 23 17:39:27.300765 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:39:27.300729    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f8f138f-b51a-4c62-9e95-47e578e25e1d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "0f8f138f-b51a-4c62-9e95-47e578e25e1d" (UID: "0f8f138f-b51a-4c62-9e95-47e578e25e1d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 17:39:27.302536 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:39:27.302509    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f8f138f-b51a-4c62-9e95-47e578e25e1d-kube-api-access-hqd6s" (OuterVolumeSpecName: "kube-api-access-hqd6s") pod "0f8f138f-b51a-4c62-9e95-47e578e25e1d" (UID: "0f8f138f-b51a-4c62-9e95-47e578e25e1d"). InnerVolumeSpecName "kube-api-access-hqd6s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 17:39:27.302643 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:39:27.302570    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f8f138f-b51a-4c62-9e95-47e578e25e1d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "0f8f138f-b51a-4c62-9e95-47e578e25e1d" (UID: "0f8f138f-b51a-4c62-9e95-47e578e25e1d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 17:39:27.401288 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:39:27.401229    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0f8f138f-b51a-4c62-9e95-47e578e25e1d-maas-api-tls\") on node \"ip-10-0-129-216.ec2.internal\" DevicePath \"\""
Jul 23 17:39:27.401288 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:39:27.401281    2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f8f138f-b51a-4c62-9e95-47e578e25e1d-openshift-service-ca\") on node \"ip-10-0-129-216.ec2.internal\" DevicePath \"\""
Jul 23 17:39:27.401288 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:39:27.401293    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hqd6s\" (UniqueName: \"kubernetes.io/projected/0f8f138f-b51a-4c62-9e95-47e578e25e1d-kube-api-access-hqd6s\") on node \"ip-10-0-129-216.ec2.internal\" DevicePath \"\""
Jul 23 17:39:27.837122 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:39:27.837085    2577 generic.go:356] "Generic (PLEG): container finished" podID="0f8f138f-b51a-4c62-9e95-47e578e25e1d" containerID="8a9685677100218be5717e689b07a3cf7d2c722752e10d04cf4ed0fe1ba99e57" exitCode=0
Jul 23 17:39:27.837327 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:39:27.837131    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7a6850d7-f9d7fcd65-5jd55" event={"ID":"0f8f138f-b51a-4c62-9e95-47e578e25e1d","Type":"ContainerDied","Data":"8a9685677100218be5717e689b07a3cf7d2c722752e10d04cf4ed0fe1ba99e57"}
Jul 23 17:39:27.837327 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:39:27.837155    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7a6850d7-f9d7fcd65-5jd55"
Jul 23 17:39:27.837327 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:39:27.837178    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7a6850d7-f9d7fcd65-5jd55" event={"ID":"0f8f138f-b51a-4c62-9e95-47e578e25e1d","Type":"ContainerDied","Data":"81967e5682556dd9f573c7375c7e320751db02832b3081eeb3575d4af6ec4117"}
Jul 23 17:39:27.837327 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:39:27.837201    2577 scope.go:117] "RemoveContainer" containerID="8a9685677100218be5717e689b07a3cf7d2c722752e10d04cf4ed0fe1ba99e57"
Jul 23 17:39:27.847140 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:39:27.847097    2577 scope.go:117] "RemoveContainer" containerID="8a9685677100218be5717e689b07a3cf7d2c722752e10d04cf4ed0fe1ba99e57"
Jul 23 17:39:27.847426 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:39:27.847404    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a9685677100218be5717e689b07a3cf7d2c722752e10d04cf4ed0fe1ba99e57\": container with ID starting with 8a9685677100218be5717e689b07a3cf7d2c722752e10d04cf4ed0fe1ba99e57 not found: ID does not exist" containerID="8a9685677100218be5717e689b07a3cf7d2c722752e10d04cf4ed0fe1ba99e57"
Jul 23 17:39:27.847525 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:39:27.847433    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a9685677100218be5717e689b07a3cf7d2c722752e10d04cf4ed0fe1ba99e57"} err="failed to get container status \"8a9685677100218be5717e689b07a3cf7d2c722752e10d04cf4ed0fe1ba99e57\": rpc error: code = NotFound desc = could not find container \"8a9685677100218be5717e689b07a3cf7d2c722752e10d04cf4ed0fe1ba99e57\": container with ID starting with 8a9685677100218be5717e689b07a3cf7d2c722752e10d04cf4ed0fe1ba99e57 not found: ID does not exist"
Jul 23 17:39:27.863448 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:39:27.863423    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7a6850d7-f9d7fcd65-5jd55"]
Jul 23 17:39:27.867689 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:39:27.867666    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7a6850d7-f9d7fcd65-5jd55"]
Jul 23 17:39:28.766488 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:39:28.766453    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0f8f138f-b51a-4c62-9e95-47e578e25e1d" path="/var/lib/kubelet/pods/0f8f138f-b51a-4c62-9e95-47e578e25e1d/volumes"
Jul 23 17:39:31.360349 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:39:31.360312    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:39:33.683015 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:39:33.682973    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:39:38.689101 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:39:38.689065    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-7a6850d7/e2e-lifecycle-model-7a6850d7-kserve-6b965cd5d7-9g9x8"]
Jul 23 17:39:38.778665 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:39:38.778629    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:39:44.182759 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:39:44.182725    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:39:44.323165 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:39:44.323125    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:39:53.709595 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:39:53.709556    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-7a6850d7/e2e-lifecycle-model-7a6850d7-kserve-6b965cd5d7-9g9x8" podUID="5b07ed6e-4258-4bce-adb0-4285527c875f" containerName="main" containerID="cri-o://a88a8b2c64d5f681bad8ea3a56853067e826047800ade08b8fdf1a189c065af4" gracePeriod=45
Jul 23 17:39:53.965143 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:39:53.965086    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-7a6850d7/e2e-lifecycle-model-7a6850d7-kserve-6b965cd5d7-9g9x8"
Jul 23 17:39:53.979932 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:39:53.979906    2577 generic.go:356] "Generic (PLEG): container finished" podID="5b07ed6e-4258-4bce-adb0-4285527c875f" containerID="a88a8b2c64d5f681bad8ea3a56853067e826047800ade08b8fdf1a189c065af4" exitCode=0
Jul 23 17:39:53.980093 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:39:53.979951    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-7a6850d7/e2e-lifecycle-model-7a6850d7-kserve-6b965cd5d7-9g9x8"
Jul 23 17:39:53.980093 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:39:53.979968    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-7a6850d7/e2e-lifecycle-model-7a6850d7-kserve-6b965cd5d7-9g9x8" event={"ID":"5b07ed6e-4258-4bce-adb0-4285527c875f","Type":"ContainerDied","Data":"a88a8b2c64d5f681bad8ea3a56853067e826047800ade08b8fdf1a189c065af4"}
Jul 23 17:39:53.980093 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:39:53.980000    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-7a6850d7/e2e-lifecycle-model-7a6850d7-kserve-6b965cd5d7-9g9x8" event={"ID":"5b07ed6e-4258-4bce-adb0-4285527c875f","Type":"ContainerDied","Data":"a989e3ee2cd9d3e9eddbbd047268489367ea709ce25ee14568f51d1bb7564a59"}
Jul 23 17:39:53.980093 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:39:53.980017    2577 scope.go:117] "RemoveContainer" containerID="a88a8b2c64d5f681bad8ea3a56853067e826047800ade08b8fdf1a189c065af4"
Jul 23 17:39:53.990355 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:39:53.990326    2577 scope.go:117] "RemoveContainer" containerID="a88a8b2c64d5f681bad8ea3a56853067e826047800ade08b8fdf1a189c065af4"
Jul 23 17:39:53.990763 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:39:53.990735    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a88a8b2c64d5f681bad8ea3a56853067e826047800ade08b8fdf1a189c065af4\": container with ID starting with a88a8b2c64d5f681bad8ea3a56853067e826047800ade08b8fdf1a189c065af4 not found: ID does not exist" containerID="a88a8b2c64d5f681bad8ea3a56853067e826047800ade08b8fdf1a189c065af4"
Jul 23 17:39:53.990880 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:39:53.990769    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a88a8b2c64d5f681bad8ea3a56853067e826047800ade08b8fdf1a189c065af4"} err="failed to get container status \"a88a8b2c64d5f681bad8ea3a56853067e826047800ade08b8fdf1a189c065af4\": rpc error: code = NotFound desc = could not find container \"a88a8b2c64d5f681bad8ea3a56853067e826047800ade08b8fdf1a189c065af4\": container with ID starting with a88a8b2c64d5f681bad8ea3a56853067e826047800ade08b8fdf1a189c065af4 not found: ID does not exist"
Jul 23 17:39:54.144529 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:39:54.144493    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b07ed6e-4258-4bce-adb0-4285527c875f-tmp-dir\") pod \"5b07ed6e-4258-4bce-adb0-4285527c875f\" (UID: \"5b07ed6e-4258-4bce-adb0-4285527c875f\") "
Jul 23 17:39:54.144728 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:39:54.144552    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b07ed6e-4258-4bce-adb0-4285527c875f-home\") pod \"5b07ed6e-4258-4bce-adb0-4285527c875f\" (UID: \"5b07ed6e-4258-4bce-adb0-4285527c875f\") "
Jul 23 17:39:54.144728 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:39:54.144597    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b07ed6e-4258-4bce-adb0-4285527c875f-model-cache\") pod \"5b07ed6e-4258-4bce-adb0-4285527c875f\" (UID: \"5b07ed6e-4258-4bce-adb0-4285527c875f\") "
Jul 23 17:39:54.144728 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:39:54.144625    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wt4fq\" (UniqueName: \"kubernetes.io/projected/5b07ed6e-4258-4bce-adb0-4285527c875f-kube-api-access-wt4fq\") pod \"5b07ed6e-4258-4bce-adb0-4285527c875f\" (UID: \"5b07ed6e-4258-4bce-adb0-4285527c875f\") "
Jul 23 17:39:54.144728 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:39:54.144643    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b07ed6e-4258-4bce-adb0-4285527c875f-tls-certs\") pod \"5b07ed6e-4258-4bce-adb0-4285527c875f\" (UID: \"5b07ed6e-4258-4bce-adb0-4285527c875f\") "
Jul 23 17:39:54.144728 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:39:54.144683    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b07ed6e-4258-4bce-adb0-4285527c875f-dshm\") pod \"5b07ed6e-4258-4bce-adb0-4285527c875f\" (UID: \"5b07ed6e-4258-4bce-adb0-4285527c875f\") "
Jul 23 17:39:54.144985 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:39:54.144809    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b07ed6e-4258-4bce-adb0-4285527c875f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5b07ed6e-4258-4bce-adb0-4285527c875f" (UID: "5b07ed6e-4258-4bce-adb0-4285527c875f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 17:39:54.144985 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:39:54.144833    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b07ed6e-4258-4bce-adb0-4285527c875f-home" (OuterVolumeSpecName: "home") pod "5b07ed6e-4258-4bce-adb0-4285527c875f" (UID: "5b07ed6e-4258-4bce-adb0-4285527c875f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 17:39:54.144985 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:39:54.144887    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b07ed6e-4258-4bce-adb0-4285527c875f-model-cache" (OuterVolumeSpecName: "model-cache") pod "5b07ed6e-4258-4bce-adb0-4285527c875f" (UID: "5b07ed6e-4258-4bce-adb0-4285527c875f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 17:39:54.145143 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:39:54.145011    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b07ed6e-4258-4bce-adb0-4285527c875f-tmp-dir\") on node \"ip-10-0-129-216.ec2.internal\" DevicePath \"\""
Jul 23 17:39:54.145143 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:39:54.145025    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b07ed6e-4258-4bce-adb0-4285527c875f-home\") on node \"ip-10-0-129-216.ec2.internal\" DevicePath \"\""
Jul 23 17:39:54.145143 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:39:54.145033    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b07ed6e-4258-4bce-adb0-4285527c875f-model-cache\") on node \"ip-10-0-129-216.ec2.internal\" DevicePath \"\""
Jul 23 17:39:54.146910 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:39:54.146885    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b07ed6e-4258-4bce-adb0-4285527c875f-dshm" (OuterVolumeSpecName: "dshm") pod "5b07ed6e-4258-4bce-adb0-4285527c875f" (UID: "5b07ed6e-4258-4bce-adb0-4285527c875f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 17:39:54.147017 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:39:54.146931    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b07ed6e-4258-4bce-adb0-4285527c875f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5b07ed6e-4258-4bce-adb0-4285527c875f" (UID: "5b07ed6e-4258-4bce-adb0-4285527c875f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 17:39:54.147017 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:39:54.146941    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b07ed6e-4258-4bce-adb0-4285527c875f-kube-api-access-wt4fq" (OuterVolumeSpecName: "kube-api-access-wt4fq") pod "5b07ed6e-4258-4bce-adb0-4285527c875f" (UID: "5b07ed6e-4258-4bce-adb0-4285527c875f"). InnerVolumeSpecName "kube-api-access-wt4fq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 17:39:54.246421 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:39:54.246334    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wt4fq\" (UniqueName: \"kubernetes.io/projected/5b07ed6e-4258-4bce-adb0-4285527c875f-kube-api-access-wt4fq\") on node \"ip-10-0-129-216.ec2.internal\" DevicePath \"\""
Jul 23 17:39:54.246421 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:39:54.246367    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b07ed6e-4258-4bce-adb0-4285527c875f-tls-certs\") on node \"ip-10-0-129-216.ec2.internal\" DevicePath \"\""
Jul 23 17:39:54.246421 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:39:54.246377    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b07ed6e-4258-4bce-adb0-4285527c875f-dshm\") on node \"ip-10-0-129-216.ec2.internal\" DevicePath \"\""
Jul 23 17:39:54.303981 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:39:54.303952    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-7a6850d7/e2e-lifecycle-model-7a6850d7-kserve-6b965cd5d7-9g9x8"]
Jul 23 17:39:54.309055 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:39:54.309032    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-7a6850d7/e2e-lifecycle-model-7a6850d7-kserve-6b965cd5d7-9g9x8"]
Jul 23 17:39:54.766607 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:39:54.766572    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5b07ed6e-4258-4bce-adb0-4285527c875f" path="/var/lib/kubelet/pods/5b07ed6e-4258-4bce-adb0-4285527c875f/volumes"
Jul 23 17:40:04.923571 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:40:04.923543    2577 scope.go:117] "RemoveContainer" containerID="f6321398abb60bf228cd37c47e83ea29ec2a0e5ed9918a16604fe48885aaec2e"
Jul 23 17:40:15.076697 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:40:15.076659    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:40:16.265105 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:40:16.265066    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:40:22.301536 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:40:22.301500    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:40:22.401613 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:40:22.401582    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:40:22.607488 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:40:22.607401    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:40:22.840474 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:40:22.840443    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:40:24.044375 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:40:24.044337    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:40:24.143639 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:40:24.143605    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:40:24.244264 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:40:24.244227    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:40:24.994118 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:40:24.994087    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:40:25.100098 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:40:25.100065    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:40:25.707420 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:40:25.707385    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:40:26.639486 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:40:26.639443    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:40:33.121159 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:40:33.121124    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:40:33.220559 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:40:33.220529    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:40:33.880754 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:40:33.880719    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:40:40.708268 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:40:40.708230    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:40:40.775492 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:40:40.775458    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:40:40.859846 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:40:40.859816    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:40:40.960206 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:40:40.960122    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:40:51.483697 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:40:51.483658    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:40:51.863243 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:40:51.863156    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:41:12.844029 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:41:12.843949    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:41:20.828967 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:41:20.828930    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:41:20.929642 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:41:20.929604    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:41:21.637880 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:41:21.637841    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:41:21.701040 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:41:21.700990    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:41:22.062879 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:41:22.062843    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:41:25.045057 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:41:25.045000    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:41:25.137689 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:41:25.137652    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:41:25.296872 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:41:25.296784    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-219aa1-79784f9856-gv8bq"]
Jul 23 17:41:25.297384 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:41:25.297359    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f8f138f-b51a-4c62-9e95-47e578e25e1d" containerName="maas-api"
Jul 23 17:41:25.297384 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:41:25.297383    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f8f138f-b51a-4c62-9e95-47e578e25e1d" containerName="maas-api"
Jul 23 17:41:25.297569 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:41:25.297397    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b07ed6e-4258-4bce-adb0-4285527c875f" containerName="main"
Jul 23 17:41:25.297569 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:41:25.297406    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b07ed6e-4258-4bce-adb0-4285527c875f" containerName="main"
Jul 23 17:41:25.297569 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:41:25.297508    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="0f8f138f-b51a-4c62-9e95-47e578e25e1d" containerName="maas-api"
Jul 23 17:41:25.297569 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:41:25.297518    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="5b07ed6e-4258-4bce-adb0-4285527c875f" containerName="main"
Jul 23 17:41:25.300960 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:41:25.300940    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-219aa1-79784f9856-gv8bq"
Jul 23 17:41:25.303217 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:41:25.303199    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-219aa1-serving-cert\""
Jul 23 17:41:25.309651 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:41:25.309628    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-219aa1-79784f9856-gv8bq"]
Jul 23 17:41:25.327543 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:41:25.327517    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:41:25.439388 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:41:25.439348    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eeff7185-99fc-4943-b860-5de616062fad-openshift-service-ca\") pod \"maas-api-e2e-inf-a-219aa1-79784f9856-gv8bq\" (UID: \"eeff7185-99fc-4943-b860-5de616062fad\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-219aa1-79784f9856-gv8bq"
Jul 23 17:41:25.439577 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:41:25.439417    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eeff7185-99fc-4943-b860-5de616062fad-maas-api-tls\") pod \"maas-api-e2e-inf-a-219aa1-79784f9856-gv8bq\" (UID: \"eeff7185-99fc-4943-b860-5de616062fad\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-219aa1-79784f9856-gv8bq"
Jul 23 17:41:25.439577 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:41:25.439486    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbnlb\" (UniqueName: \"kubernetes.io/projected/eeff7185-99fc-4943-b860-5de616062fad-kube-api-access-tbnlb\") pod \"maas-api-e2e-inf-a-219aa1-79784f9856-gv8bq\" (UID: \"eeff7185-99fc-4943-b860-5de616062fad\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-219aa1-79784f9856-gv8bq"
Jul 23 17:41:25.520246 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:41:25.520208    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:41:25.540645 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:41:25.540612    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eeff7185-99fc-4943-b860-5de616062fad-openshift-service-ca\") pod \"maas-api-e2e-inf-a-219aa1-79784f9856-gv8bq\" (UID: \"eeff7185-99fc-4943-b860-5de616062fad\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-219aa1-79784f9856-gv8bq"
Jul 23 17:41:25.540810 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:41:25.540677    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eeff7185-99fc-4943-b860-5de616062fad-maas-api-tls\") pod \"maas-api-e2e-inf-a-219aa1-79784f9856-gv8bq\" (UID: \"eeff7185-99fc-4943-b860-5de616062fad\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-219aa1-79784f9856-gv8bq"
Jul 23 17:41:25.540810 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:41:25.540743    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbnlb\" (UniqueName: \"kubernetes.io/projected/eeff7185-99fc-4943-b860-5de616062fad-kube-api-access-tbnlb\") pod \"maas-api-e2e-inf-a-219aa1-79784f9856-gv8bq\" (UID: \"eeff7185-99fc-4943-b860-5de616062fad\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-219aa1-79784f9856-gv8bq"
Jul 23 17:41:25.541306 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:41:25.541284    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eeff7185-99fc-4943-b860-5de616062fad-openshift-service-ca\") pod \"maas-api-e2e-inf-a-219aa1-79784f9856-gv8bq\" (UID: \"eeff7185-99fc-4943-b860-5de616062fad\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-219aa1-79784f9856-gv8bq"
Jul 23 17:41:25.543215 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:41:25.543196    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eeff7185-99fc-4943-b860-5de616062fad-maas-api-tls\") pod \"maas-api-e2e-inf-a-219aa1-79784f9856-gv8bq\" (UID: \"eeff7185-99fc-4943-b860-5de616062fad\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-219aa1-79784f9856-gv8bq"
Jul 23 17:41:25.548840 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:41:25.548781    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbnlb\" (UniqueName: \"kubernetes.io/projected/eeff7185-99fc-4943-b860-5de616062fad-kube-api-access-tbnlb\") pod \"maas-api-e2e-inf-a-219aa1-79784f9856-gv8bq\" (UID: \"eeff7185-99fc-4943-b860-5de616062fad\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-219aa1-79784f9856-gv8bq"
Jul 23 17:41:25.614499 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:41:25.614460    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-219aa1-79784f9856-gv8bq"
Jul 23 17:41:25.755740 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:41:25.755715    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-219aa1-79784f9856-gv8bq"]
Jul 23 17:41:25.757489 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:41:25.757463    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeeff7185_99fc_4943_b860_5de616062fad.slice/crio-dddd317ef5fe60d2a52549ee1c47853c5c617f30323bbcb12117d13903f9d40c WatchSource:0}: Error finding container dddd317ef5fe60d2a52549ee1c47853c5c617f30323bbcb12117d13903f9d40c: Status 404 returned error can't find the container with id dddd317ef5fe60d2a52549ee1c47853c5c617f30323bbcb12117d13903f9d40c
Jul 23 17:41:25.789445 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:41:25.789416    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:41:25.892606 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:41:25.892508    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:41:26.483459 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:41:26.483416    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-219aa1-79784f9856-gv8bq" event={"ID":"eeff7185-99fc-4943-b860-5de616062fad","Type":"ContainerStarted","Data":"232472ffe4a806eb204c85bee6850884dd50481af548013b5ace32b31e4fa76b"}
Jul 23 17:41:26.483849 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:41:26.483510    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-219aa1-79784f9856-gv8bq"
Jul 23 17:41:26.483849 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:41:26.483532    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-219aa1-79784f9856-gv8bq" event={"ID":"eeff7185-99fc-4943-b860-5de616062fad","Type":"ContainerStarted","Data":"dddd317ef5fe60d2a52549ee1c47853c5c617f30323bbcb12117d13903f9d40c"}
Jul 23 17:41:26.508286 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:41:26.508184    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-219aa1-79784f9856-gv8bq" podStartSLOduration=1.01258742 podStartE2EDuration="1.508169279s" totalImagesPullingTime="495.581859ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 17:41:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 17:41:26.505628396 +0000 UTC m=+2182.275913917" watchObservedRunningTime="2026-07-23 17:41:26.508169279 +0000 UTC m=+2182.278454797"
Jul 23 17:41:27.206196 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:41:27.206162    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:41:29.535981 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:41:29.535950    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:41:29.898070 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:41:29.897988    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:41:30.227360 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:41:30.227318    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:41:30.443779 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:41:30.443748    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:41:32.494604 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:41:32.494572    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-219aa1-79784f9856-gv8bq"
Jul 23 17:41:33.024963 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:41:33.024926    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:41:33.115078 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:41:33.115036    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:41:33.131938 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:41:33.131907    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-7363d9-7cc4448956-7h447"]
Jul 23 17:41:33.136576 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:41:33.136552    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7363d9-7cc4448956-7h447"
Jul 23 17:41:33.139527 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:41:33.139506    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-7363d9-serving-cert\""
Jul 23 17:41:33.149626 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:41:33.149599    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-7363d9-7cc4448956-7h447"]
Jul 23 17:41:33.194644 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:41:33.194608    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:41:33.207804 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:41:33.207773    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f79ebd27-484a-4f75-915e-7ef4ba952817-maas-api-tls\") pod \"maas-api-e2e-inf-b-7363d9-7cc4448956-7h447\" (UID: \"f79ebd27-484a-4f75-915e-7ef4ba952817\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7363d9-7cc4448956-7h447"
Jul 23 17:41:33.207917 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:41:33.207807    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f79ebd27-484a-4f75-915e-7ef4ba952817-openshift-service-ca\") pod \"maas-api-e2e-inf-b-7363d9-7cc4448956-7h447\" (UID: \"f79ebd27-484a-4f75-915e-7ef4ba952817\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7363d9-7cc4448956-7h447"
Jul 23 17:41:33.207982 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:41:33.207961    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5k6wn\" (UniqueName: \"kubernetes.io/projected/f79ebd27-484a-4f75-915e-7ef4ba952817-kube-api-access-5k6wn\") pod \"maas-api-e2e-inf-b-7363d9-7cc4448956-7h447\" (UID: \"f79ebd27-484a-4f75-915e-7ef4ba952817\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7363d9-7cc4448956-7h447"
Jul 23 17:41:33.309015 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:41:33.308922    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5k6wn\" (UniqueName: \"kubernetes.io/projected/f79ebd27-484a-4f75-915e-7ef4ba952817-kube-api-access-5k6wn\") pod \"maas-api-e2e-inf-b-7363d9-7cc4448956-7h447\" (UID: \"f79ebd27-484a-4f75-915e-7ef4ba952817\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7363d9-7cc4448956-7h447"
Jul 23 17:41:33.309015 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:41:33.308976    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f79ebd27-484a-4f75-915e-7ef4ba952817-maas-api-tls\") pod \"maas-api-e2e-inf-b-7363d9-7cc4448956-7h447\" (UID: \"f79ebd27-484a-4f75-915e-7ef4ba952817\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7363d9-7cc4448956-7h447"
Jul 23 17:41:33.309015 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:41:33.309003    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f79ebd27-484a-4f75-915e-7ef4ba952817-openshift-service-ca\") pod \"maas-api-e2e-inf-b-7363d9-7cc4448956-7h447\" (UID: \"f79ebd27-484a-4f75-915e-7ef4ba952817\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7363d9-7cc4448956-7h447"
Jul 23 17:41:33.309362 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:41:33.309130    2577 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-7363d9-serving-cert: secret "maas-api-e2e-inf-b-7363d9-serving-cert" not found
Jul 23 17:41:33.309362 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:41:33.309216    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f79ebd27-484a-4f75-915e-7ef4ba952817-maas-api-tls podName:f79ebd27-484a-4f75-915e-7ef4ba952817 nodeName:}" failed. No retries permitted until 2026-07-23 17:41:33.809199578 +0000 UTC m=+2189.579485079 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f79ebd27-484a-4f75-915e-7ef4ba952817-maas-api-tls") pod "maas-api-e2e-inf-b-7363d9-7cc4448956-7h447" (UID: "f79ebd27-484a-4f75-915e-7ef4ba952817") : secret "maas-api-e2e-inf-b-7363d9-serving-cert" not found
Jul 23 17:41:33.309744 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:41:33.309722    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f79ebd27-484a-4f75-915e-7ef4ba952817-openshift-service-ca\") pod \"maas-api-e2e-inf-b-7363d9-7cc4448956-7h447\" (UID: \"f79ebd27-484a-4f75-915e-7ef4ba952817\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7363d9-7cc4448956-7h447"
Jul 23 17:41:33.317113 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:41:33.317087    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5k6wn\" (UniqueName: \"kubernetes.io/projected/f79ebd27-484a-4f75-915e-7ef4ba952817-kube-api-access-5k6wn\") pod \"maas-api-e2e-inf-b-7363d9-7cc4448956-7h447\" (UID: \"f79ebd27-484a-4f75-915e-7ef4ba952817\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7363d9-7cc4448956-7h447"
Jul 23 17:41:33.741115 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:41:33.741079    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:41:33.814604 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:41:33.814567    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f79ebd27-484a-4f75-915e-7ef4ba952817-maas-api-tls\") pod \"maas-api-e2e-inf-b-7363d9-7cc4448956-7h447\" (UID: \"f79ebd27-484a-4f75-915e-7ef4ba952817\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7363d9-7cc4448956-7h447"
Jul 23 17:41:33.814786 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:41:33.814709    2577 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-7363d9-serving-cert: secret "maas-api-e2e-inf-b-7363d9-serving-cert" not found
Jul 23 17:41:33.814786 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:41:33.814786    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f79ebd27-484a-4f75-915e-7ef4ba952817-maas-api-tls podName:f79ebd27-484a-4f75-915e-7ef4ba952817 nodeName:}" failed. No retries permitted until 2026-07-23 17:41:34.814770928 +0000 UTC m=+2190.585056424 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f79ebd27-484a-4f75-915e-7ef4ba952817-maas-api-tls") pod "maas-api-e2e-inf-b-7363d9-7cc4448956-7h447" (UID: "f79ebd27-484a-4f75-915e-7ef4ba952817") : secret "maas-api-e2e-inf-b-7363d9-serving-cert" not found
Jul 23 17:41:33.886865 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:41:33.886828    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:41:34.109377 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:41:34.109298    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:41:34.824275 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:41:34.824216    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f79ebd27-484a-4f75-915e-7ef4ba952817-maas-api-tls\") pod \"maas-api-e2e-inf-b-7363d9-7cc4448956-7h447\" (UID: \"f79ebd27-484a-4f75-915e-7ef4ba952817\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7363d9-7cc4448956-7h447"
Jul 23 17:41:34.826642 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:41:34.826615    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f79ebd27-484a-4f75-915e-7ef4ba952817-maas-api-tls\") pod \"maas-api-e2e-inf-b-7363d9-7cc4448956-7h447\" (UID: \"f79ebd27-484a-4f75-915e-7ef4ba952817\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7363d9-7cc4448956-7h447"
Jul 23 17:41:34.950387 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:41:34.950353    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7363d9-7cc4448956-7h447"
Jul 23 17:41:35.091407 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:41:35.091380    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-7363d9-7cc4448956-7h447"]
Jul 23 17:41:35.093646 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:41:35.093620    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf79ebd27_484a_4f75_915e_7ef4ba952817.slice/crio-7e0423be5503ac88fc20c8831eafed50dcfeb208329b75b7483e899b51334953 WatchSource:0}: Error finding container 7e0423be5503ac88fc20c8831eafed50dcfeb208329b75b7483e899b51334953: Status 404 returned error can't find the container with id 7e0423be5503ac88fc20c8831eafed50dcfeb208329b75b7483e899b51334953
Jul 23 17:41:35.536129 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:41:35.536074    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7363d9-7cc4448956-7h447" event={"ID":"f79ebd27-484a-4f75-915e-7ef4ba952817","Type":"ContainerStarted","Data":"7e0423be5503ac88fc20c8831eafed50dcfeb208329b75b7483e899b51334953"}
Jul 23 17:41:36.543690 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:41:36.543655    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7363d9-7cc4448956-7h447" event={"ID":"f79ebd27-484a-4f75-915e-7ef4ba952817","Type":"ContainerStarted","Data":"676b601a59919ef432139db31bf4c5a4082f111f0793d998fa89f3c54e965800"}
Jul 23 17:41:36.544197 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:41:36.543762    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7363d9-7cc4448956-7h447"
Jul 23 17:41:36.560121 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:41:36.560075    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7363d9-7cc4448956-7h447" podStartSLOduration=3.046283433 podStartE2EDuration="3.560060958s" totalImagesPullingTime="513.777525ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 17:41:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 17:41:36.5592802 +0000 UTC m=+2192.329565716" watchObservedRunningTime="2026-07-23 17:41:36.560060958 +0000 UTC m=+2192.330346537"
Jul 23 17:41:37.592777 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:41:37.592740    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:41:42.553896 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:41:42.553866    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7363d9-7cc4448956-7h447"
Jul 23 17:41:42.799267 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:41:42.799228    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-219aa1/test-model-219aa1-kserve-6b64757995-xnntf"]
Jul 23 17:41:42.803123 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:41:42.803101    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-219aa1/test-model-219aa1-kserve-6b64757995-xnntf"
Jul 23 17:41:42.805879 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:41:42.805821    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-219aa1\"/\"openshift-service-ca.crt\""
Jul 23 17:41:42.805879 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:41:42.805843    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-219aa1\"/\"kube-root-ca.crt\""
Jul 23 17:41:42.806064 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:41:42.805916    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-219aa1\"/\"test-model-219aa1-kserve-self-signed-certs\""
Jul 23 17:41:42.806064 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:41:42.806052    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-219aa1\"/\"default-dockercfg-mkbjj\""
Jul 23 17:41:42.812897 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:41:42.812875    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-219aa1/test-model-219aa1-kserve-6b64757995-xnntf"]
Jul 23 17:41:42.830374 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:41:42.830346    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:41:42.898813 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:41:42.898780    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba76055b-2a37-43eb-96f9-1d95c1edabd1-home\") pod \"test-model-219aa1-kserve-6b64757995-xnntf\" (UID: \"ba76055b-2a37-43eb-96f9-1d95c1edabd1\") " pod="ai-tenant-e2e-inf-a-219aa1/test-model-219aa1-kserve-6b64757995-xnntf"
Jul 23 17:41:42.898813 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:41:42.898820    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba76055b-2a37-43eb-96f9-1d95c1edabd1-model-cache\") pod \"test-model-219aa1-kserve-6b64757995-xnntf\" (UID: \"ba76055b-2a37-43eb-96f9-1d95c1edabd1\") " pod="ai-tenant-e2e-inf-a-219aa1/test-model-219aa1-kserve-6b64757995-xnntf"
Jul 23 17:41:42.899106 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:41:42.898852    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sjgtr\" (UniqueName: \"kubernetes.io/projected/ba76055b-2a37-43eb-96f9-1d95c1edabd1-kube-api-access-sjgtr\") pod \"test-model-219aa1-kserve-6b64757995-xnntf\" (UID: \"ba76055b-2a37-43eb-96f9-1d95c1edabd1\") " pod="ai-tenant-e2e-inf-a-219aa1/test-model-219aa1-kserve-6b64757995-xnntf"
Jul 23 17:41:42.899106 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:41:42.898949    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba76055b-2a37-43eb-96f9-1d95c1edabd1-tls-certs\") pod \"test-model-219aa1-kserve-6b64757995-xnntf\" (UID: \"ba76055b-2a37-43eb-96f9-1d95c1edabd1\") " pod="ai-tenant-e2e-inf-a-219aa1/test-model-219aa1-kserve-6b64757995-xnntf"
Jul 23 17:41:42.899106 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:41:42.898980    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba76055b-2a37-43eb-96f9-1d95c1edabd1-dshm\") pod \"test-model-219aa1-kserve-6b64757995-xnntf\" (UID: \"ba76055b-2a37-43eb-96f9-1d95c1edabd1\") " pod="ai-tenant-e2e-inf-a-219aa1/test-model-219aa1-kserve-6b64757995-xnntf"
Jul 23 17:41:42.899106 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:41:42.899004    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba76055b-2a37-43eb-96f9-1d95c1edabd1-tmp-dir\") pod \"test-model-219aa1-kserve-6b64757995-xnntf\" (UID: \"ba76055b-2a37-43eb-96f9-1d95c1edabd1\") " pod="ai-tenant-e2e-inf-a-219aa1/test-model-219aa1-kserve-6b64757995-xnntf"
Jul 23 17:41:42.999587 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:41:42.999549    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba76055b-2a37-43eb-96f9-1d95c1edabd1-tls-certs\") pod \"test-model-219aa1-kserve-6b64757995-xnntf\" (UID: \"ba76055b-2a37-43eb-96f9-1d95c1edabd1\") " pod="ai-tenant-e2e-inf-a-219aa1/test-model-219aa1-kserve-6b64757995-xnntf"
Jul 23 17:41:42.999777 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:41:42.999604    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba76055b-2a37-43eb-96f9-1d95c1edabd1-dshm\") pod \"test-model-219aa1-kserve-6b64757995-xnntf\" (UID: \"ba76055b-2a37-43eb-96f9-1d95c1edabd1\") " pod="ai-tenant-e2e-inf-a-219aa1/test-model-219aa1-kserve-6b64757995-xnntf"
Jul 23 17:41:42.999777 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:41:42.999722    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba76055b-2a37-43eb-96f9-1d95c1edabd1-tmp-dir\") pod \"test-model-219aa1-kserve-6b64757995-xnntf\" (UID: \"ba76055b-2a37-43eb-96f9-1d95c1edabd1\") " pod="ai-tenant-e2e-inf-a-219aa1/test-model-219aa1-kserve-6b64757995-xnntf"
Jul 23 17:41:42.999900 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:41:42.999874    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba76055b-2a37-43eb-96f9-1d95c1edabd1-home\") pod \"test-model-219aa1-kserve-6b64757995-xnntf\" (UID: \"ba76055b-2a37-43eb-96f9-1d95c1edabd1\") " pod="ai-tenant-e2e-inf-a-219aa1/test-model-219aa1-kserve-6b64757995-xnntf"
Jul 23 17:41:42.999958 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:41:42.999902    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba76055b-2a37-43eb-96f9-1d95c1edabd1-model-cache\") pod \"test-model-219aa1-kserve-6b64757995-xnntf\" (UID: \"ba76055b-2a37-43eb-96f9-1d95c1edabd1\") " pod="ai-tenant-e2e-inf-a-219aa1/test-model-219aa1-kserve-6b64757995-xnntf"
Jul 23 17:41:42.999958 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:41:42.999933    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sjgtr\" (UniqueName: \"kubernetes.io/projected/ba76055b-2a37-43eb-96f9-1d95c1edabd1-kube-api-access-sjgtr\") pod \"test-model-219aa1-kserve-6b64757995-xnntf\" (UID: \"ba76055b-2a37-43eb-96f9-1d95c1edabd1\") " pod="ai-tenant-e2e-inf-a-219aa1/test-model-219aa1-kserve-6b64757995-xnntf"
Jul 23 17:41:43.000127 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:41:43.000083    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba76055b-2a37-43eb-96f9-1d95c1edabd1-tmp-dir\") pod \"test-model-219aa1-kserve-6b64757995-xnntf\" (UID: \"ba76055b-2a37-43eb-96f9-1d95c1edabd1\") " pod="ai-tenant-e2e-inf-a-219aa1/test-model-219aa1-kserve-6b64757995-xnntf"
Jul 23 17:41:43.000290 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:41:43.000178    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba76055b-2a37-43eb-96f9-1d95c1edabd1-home\") pod \"test-model-219aa1-kserve-6b64757995-xnntf\" (UID: \"ba76055b-2a37-43eb-96f9-1d95c1edabd1\") " pod="ai-tenant-e2e-inf-a-219aa1/test-model-219aa1-kserve-6b64757995-xnntf"
Jul 23 17:41:43.000290 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:41:43.000246    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba76055b-2a37-43eb-96f9-1d95c1edabd1-model-cache\") pod \"test-model-219aa1-kserve-6b64757995-xnntf\" (UID: \"ba76055b-2a37-43eb-96f9-1d95c1edabd1\") " pod="ai-tenant-e2e-inf-a-219aa1/test-model-219aa1-kserve-6b64757995-xnntf"
Jul 23 17:41:43.001926 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:41:43.001903    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba76055b-2a37-43eb-96f9-1d95c1edabd1-dshm\") pod \"test-model-219aa1-kserve-6b64757995-xnntf\" (UID: \"ba76055b-2a37-43eb-96f9-1d95c1edabd1\") " pod="ai-tenant-e2e-inf-a-219aa1/test-model-219aa1-kserve-6b64757995-xnntf"
Jul 23 17:41:43.002088 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:41:43.002071    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba76055b-2a37-43eb-96f9-1d95c1edabd1-tls-certs\") pod \"test-model-219aa1-kserve-6b64757995-xnntf\" (UID: \"ba76055b-2a37-43eb-96f9-1d95c1edabd1\") " pod="ai-tenant-e2e-inf-a-219aa1/test-model-219aa1-kserve-6b64757995-xnntf"
Jul 23 17:41:43.008984 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:41:43.008960    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sjgtr\" (UniqueName: \"kubernetes.io/projected/ba76055b-2a37-43eb-96f9-1d95c1edabd1-kube-api-access-sjgtr\") pod \"test-model-219aa1-kserve-6b64757995-xnntf\" (UID: \"ba76055b-2a37-43eb-96f9-1d95c1edabd1\") " pod="ai-tenant-e2e-inf-a-219aa1/test-model-219aa1-kserve-6b64757995-xnntf"
Jul 23 17:41:43.113026 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:41:43.112939    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-219aa1/test-model-219aa1-kserve-6b64757995-xnntf"
Jul 23 17:41:43.255595 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:41:43.255567    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-219aa1/test-model-219aa1-kserve-6b64757995-xnntf"]
Jul 23 17:41:43.583803 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:41:43.583769    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-219aa1/test-model-219aa1-kserve-6b64757995-xnntf" event={"ID":"ba76055b-2a37-43eb-96f9-1d95c1edabd1","Type":"ContainerStarted","Data":"50a16d908515fb7b399b9dc6041ecb82c20cb78abd2158cc1d1a74326184a9e8"}
Jul 23 17:41:43.583803 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:41:43.583809    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-219aa1/test-model-219aa1-kserve-6b64757995-xnntf" event={"ID":"ba76055b-2a37-43eb-96f9-1d95c1edabd1","Type":"ContainerStarted","Data":"a2ce9d7a44fa1ec32c198017fd55f18b975fe99946e543bedefb141a6875a29b"}
Jul 23 17:41:43.606417 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:41:43.606363    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-219aa1/test-model-219aa1-kserve-6b64757995-xnntf" podStartSLOduration=1.6063492780000002 podStartE2EDuration="1.606349278s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 17:41:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 17:41:43.605885877 +0000 UTC m=+2199.376171398" watchObservedRunningTime="2026-07-23 17:41:43.606349278 +0000 UTC m=+2199.376634796"
Jul 23 17:41:47.884533 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:41:47.884498    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:41:47.981649 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:41:47.981616    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:41:53.113394 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:41:53.113350    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-219aa1/test-model-219aa1-kserve-6b64757995-xnntf"
Jul 23 17:41:53.113968 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:41:53.113411    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-219aa1/test-model-219aa1-kserve-6b64757995-xnntf"
Jul 23 17:41:53.125684 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:41:53.125657    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-219aa1/test-model-219aa1-kserve-6b64757995-xnntf"
Jul 23 17:41:53.654434 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:41:53.654405    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-219aa1/test-model-219aa1-kserve-6b64757995-xnntf"
Jul 23 17:41:56.356401 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:41:56.356359    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:41:57.893826 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:41:57.893792    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:42:04.963087 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:42:04.963054    2577 scope.go:117] "RemoveContainer" containerID="5e9089d0b3c360d28d7d3420648d870a111489f9bcd820108be5eca771300ede"
Jul 23 17:42:06.973568 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:42:06.973530    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:42:07.078899 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:42:07.078848    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:42:37.626543 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:42:37.626509    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-219aa1/test-model-219aa1-kserve-6b64757995-xnntf"]
Jul 23 17:42:37.892012 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:42:37.891920    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:42:38.753056 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:42:38.753018    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:42:39.693607 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:42:39.693571    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:42:39.799603 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:42:39.799564    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:42:49.879676 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:42:49.879642    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-219aa1-79784f9856-gv8bq"]
Jul 23 17:42:49.880132 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:42:49.879906    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-219aa1-79784f9856-gv8bq" podUID="eeff7185-99fc-4943-b860-5de616062fad" containerName="maas-api" containerID="cri-o://232472ffe4a806eb204c85bee6850884dd50481af548013b5ace32b31e4fa76b" gracePeriod=30
Jul 23 17:42:49.921040 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:42:49.921014    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:42:50.139630 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:42:50.139572    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-219aa1-79784f9856-gv8bq"
Jul 23 17:42:50.211241 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:42:50.211206    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eeff7185-99fc-4943-b860-5de616062fad-openshift-service-ca\") pod \"eeff7185-99fc-4943-b860-5de616062fad\" (UID: \"eeff7185-99fc-4943-b860-5de616062fad\") "
Jul 23 17:42:50.211416 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:42:50.211317    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eeff7185-99fc-4943-b860-5de616062fad-maas-api-tls\") pod \"eeff7185-99fc-4943-b860-5de616062fad\" (UID: \"eeff7185-99fc-4943-b860-5de616062fad\") "
Jul 23 17:42:50.211416 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:42:50.211345    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tbnlb\" (UniqueName: \"kubernetes.io/projected/eeff7185-99fc-4943-b860-5de616062fad-kube-api-access-tbnlb\") pod \"eeff7185-99fc-4943-b860-5de616062fad\" (UID: \"eeff7185-99fc-4943-b860-5de616062fad\") "
Jul 23 17:42:50.211628 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:42:50.211607    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eeff7185-99fc-4943-b860-5de616062fad-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "eeff7185-99fc-4943-b860-5de616062fad" (UID: "eeff7185-99fc-4943-b860-5de616062fad"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 17:42:50.213431 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:42:50.213397    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eeff7185-99fc-4943-b860-5de616062fad-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "eeff7185-99fc-4943-b860-5de616062fad" (UID: "eeff7185-99fc-4943-b860-5de616062fad"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 17:42:50.213545 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:42:50.213469    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eeff7185-99fc-4943-b860-5de616062fad-kube-api-access-tbnlb" (OuterVolumeSpecName: "kube-api-access-tbnlb") pod "eeff7185-99fc-4943-b860-5de616062fad" (UID: "eeff7185-99fc-4943-b860-5de616062fad"). InnerVolumeSpecName "kube-api-access-tbnlb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 17:42:50.312730 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:42:50.312692    2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eeff7185-99fc-4943-b860-5de616062fad-openshift-service-ca\") on node \"ip-10-0-129-216.ec2.internal\" DevicePath \"\""
Jul 23 17:42:50.312730 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:42:50.312725    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eeff7185-99fc-4943-b860-5de616062fad-maas-api-tls\") on node \"ip-10-0-129-216.ec2.internal\" DevicePath \"\""
Jul 23 17:42:50.312928 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:42:50.312740    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tbnlb\" (UniqueName: \"kubernetes.io/projected/eeff7185-99fc-4943-b860-5de616062fad-kube-api-access-tbnlb\") on node \"ip-10-0-129-216.ec2.internal\" DevicePath \"\""
Jul 23 17:42:50.961964 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:42:50.961921    2577 generic.go:356] "Generic (PLEG): container finished" podID="eeff7185-99fc-4943-b860-5de616062fad" containerID="232472ffe4a806eb204c85bee6850884dd50481af548013b5ace32b31e4fa76b" exitCode=0
Jul 23 17:42:50.962382 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:42:50.961991    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-219aa1-79784f9856-gv8bq"
Jul 23 17:42:50.962382 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:42:50.962002    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-219aa1-79784f9856-gv8bq" event={"ID":"eeff7185-99fc-4943-b860-5de616062fad","Type":"ContainerDied","Data":"232472ffe4a806eb204c85bee6850884dd50481af548013b5ace32b31e4fa76b"}
Jul 23 17:42:50.962382 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:42:50.962062    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-219aa1-79784f9856-gv8bq" event={"ID":"eeff7185-99fc-4943-b860-5de616062fad","Type":"ContainerDied","Data":"dddd317ef5fe60d2a52549ee1c47853c5c617f30323bbcb12117d13903f9d40c"}
Jul 23 17:42:50.962382 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:42:50.962093    2577 scope.go:117] "RemoveContainer" containerID="232472ffe4a806eb204c85bee6850884dd50481af548013b5ace32b31e4fa76b"
Jul 23 17:42:50.971411 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:42:50.971335    2577 scope.go:117] "RemoveContainer" containerID="232472ffe4a806eb204c85bee6850884dd50481af548013b5ace32b31e4fa76b"
Jul 23 17:42:50.971606 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:42:50.971586    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"232472ffe4a806eb204c85bee6850884dd50481af548013b5ace32b31e4fa76b\": container with ID starting with 232472ffe4a806eb204c85bee6850884dd50481af548013b5ace32b31e4fa76b not found: ID does not exist" containerID="232472ffe4a806eb204c85bee6850884dd50481af548013b5ace32b31e4fa76b"
Jul 23 17:42:50.971673 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:42:50.971618    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"232472ffe4a806eb204c85bee6850884dd50481af548013b5ace32b31e4fa76b"} err="failed to get container status \"232472ffe4a806eb204c85bee6850884dd50481af548013b5ace32b31e4fa76b\": rpc error: code = NotFound desc = could not find container \"232472ffe4a806eb204c85bee6850884dd50481af548013b5ace32b31e4fa76b\": container with ID starting with 232472ffe4a806eb204c85bee6850884dd50481af548013b5ace32b31e4fa76b not found: ID does not exist"
Jul 23 17:42:50.981752 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:42:50.981731    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-219aa1-79784f9856-gv8bq"]
Jul 23 17:42:50.985606 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:42:50.985578    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-219aa1-79784f9856-gv8bq"]
Jul 23 17:42:52.648140 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:42:52.648094    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-219aa1/test-model-219aa1-kserve-6b64757995-xnntf" podUID="ba76055b-2a37-43eb-96f9-1d95c1edabd1" containerName="main" containerID="cri-o://50a16d908515fb7b399b9dc6041ecb82c20cb78abd2158cc1d1a74326184a9e8" gracePeriod=45
Jul 23 17:42:52.766226 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:42:52.766201    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eeff7185-99fc-4943-b860-5de616062fad" path="/var/lib/kubelet/pods/eeff7185-99fc-4943-b860-5de616062fad/volumes"
Jul 23 17:42:52.911301 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:42:52.911236    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-219aa1/test-model-219aa1-kserve-6b64757995-xnntf"
Jul 23 17:42:52.932527 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:42:52.932500    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba76055b-2a37-43eb-96f9-1d95c1edabd1-dshm\") pod \"ba76055b-2a37-43eb-96f9-1d95c1edabd1\" (UID: \"ba76055b-2a37-43eb-96f9-1d95c1edabd1\") "
Jul 23 17:42:52.932676 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:42:52.932546    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba76055b-2a37-43eb-96f9-1d95c1edabd1-home\") pod \"ba76055b-2a37-43eb-96f9-1d95c1edabd1\" (UID: \"ba76055b-2a37-43eb-96f9-1d95c1edabd1\") "
Jul 23 17:42:52.932676 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:42:52.932584    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba76055b-2a37-43eb-96f9-1d95c1edabd1-model-cache\") pod \"ba76055b-2a37-43eb-96f9-1d95c1edabd1\" (UID: \"ba76055b-2a37-43eb-96f9-1d95c1edabd1\") "
Jul 23 17:42:52.932676 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:42:52.932611    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba76055b-2a37-43eb-96f9-1d95c1edabd1-tls-certs\") pod \"ba76055b-2a37-43eb-96f9-1d95c1edabd1\" (UID: \"ba76055b-2a37-43eb-96f9-1d95c1edabd1\") "
Jul 23 17:42:52.932676 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:42:52.932655    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sjgtr\" (UniqueName: \"kubernetes.io/projected/ba76055b-2a37-43eb-96f9-1d95c1edabd1-kube-api-access-sjgtr\") pod \"ba76055b-2a37-43eb-96f9-1d95c1edabd1\" (UID: \"ba76055b-2a37-43eb-96f9-1d95c1edabd1\") "
Jul 23 17:42:52.932912 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:42:52.932720    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba76055b-2a37-43eb-96f9-1d95c1edabd1-tmp-dir\") pod \"ba76055b-2a37-43eb-96f9-1d95c1edabd1\" (UID: \"ba76055b-2a37-43eb-96f9-1d95c1edabd1\") "
Jul 23 17:42:52.933051 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:42:52.933025    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba76055b-2a37-43eb-96f9-1d95c1edabd1-model-cache" (OuterVolumeSpecName: "model-cache") pod "ba76055b-2a37-43eb-96f9-1d95c1edabd1" (UID: "ba76055b-2a37-43eb-96f9-1d95c1edabd1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 17:42:52.933273 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:42:52.933232    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba76055b-2a37-43eb-96f9-1d95c1edabd1-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ba76055b-2a37-43eb-96f9-1d95c1edabd1" (UID: "ba76055b-2a37-43eb-96f9-1d95c1edabd1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 17:42:52.933521 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:42:52.933495    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba76055b-2a37-43eb-96f9-1d95c1edabd1-home" (OuterVolumeSpecName: "home") pod "ba76055b-2a37-43eb-96f9-1d95c1edabd1" (UID: "ba76055b-2a37-43eb-96f9-1d95c1edabd1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 17:42:52.935883 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:42:52.935858    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba76055b-2a37-43eb-96f9-1d95c1edabd1-dshm" (OuterVolumeSpecName: "dshm") pod "ba76055b-2a37-43eb-96f9-1d95c1edabd1" (UID: "ba76055b-2a37-43eb-96f9-1d95c1edabd1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 17:42:52.935988 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:42:52.935948    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba76055b-2a37-43eb-96f9-1d95c1edabd1-kube-api-access-sjgtr" (OuterVolumeSpecName: "kube-api-access-sjgtr") pod "ba76055b-2a37-43eb-96f9-1d95c1edabd1" (UID: "ba76055b-2a37-43eb-96f9-1d95c1edabd1"). InnerVolumeSpecName "kube-api-access-sjgtr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 17:42:52.936877 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:42:52.936857    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ba76055b-2a37-43eb-96f9-1d95c1edabd1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ba76055b-2a37-43eb-96f9-1d95c1edabd1" (UID: "ba76055b-2a37-43eb-96f9-1d95c1edabd1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 17:42:52.974816 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:42:52.974780    2577 generic.go:356] "Generic (PLEG): container finished" podID="ba76055b-2a37-43eb-96f9-1d95c1edabd1" containerID="50a16d908515fb7b399b9dc6041ecb82c20cb78abd2158cc1d1a74326184a9e8" exitCode=0
Jul 23 17:42:52.974924 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:42:52.974827    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-219aa1/test-model-219aa1-kserve-6b64757995-xnntf"
Jul 23 17:42:52.974924 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:42:52.974865    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-219aa1/test-model-219aa1-kserve-6b64757995-xnntf" event={"ID":"ba76055b-2a37-43eb-96f9-1d95c1edabd1","Type":"ContainerDied","Data":"50a16d908515fb7b399b9dc6041ecb82c20cb78abd2158cc1d1a74326184a9e8"}
Jul 23 17:42:52.974924 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:42:52.974911    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-219aa1/test-model-219aa1-kserve-6b64757995-xnntf" event={"ID":"ba76055b-2a37-43eb-96f9-1d95c1edabd1","Type":"ContainerDied","Data":"a2ce9d7a44fa1ec32c198017fd55f18b975fe99946e543bedefb141a6875a29b"}
Jul 23 17:42:52.975036 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:42:52.974930    2577 scope.go:117] "RemoveContainer" containerID="50a16d908515fb7b399b9dc6041ecb82c20cb78abd2158cc1d1a74326184a9e8"
Jul 23 17:42:52.984536 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:42:52.984512    2577 scope.go:117] "RemoveContainer" containerID="50a16d908515fb7b399b9dc6041ecb82c20cb78abd2158cc1d1a74326184a9e8"
Jul 23 17:42:52.984807 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:42:52.984787    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50a16d908515fb7b399b9dc6041ecb82c20cb78abd2158cc1d1a74326184a9e8\": container with ID starting with 50a16d908515fb7b399b9dc6041ecb82c20cb78abd2158cc1d1a74326184a9e8 not found: ID does not exist" containerID="50a16d908515fb7b399b9dc6041ecb82c20cb78abd2158cc1d1a74326184a9e8"
Jul 23 17:42:52.984875 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:42:52.984815    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50a16d908515fb7b399b9dc6041ecb82c20cb78abd2158cc1d1a74326184a9e8"} err="failed to get container status \"50a16d908515fb7b399b9dc6041ecb82c20cb78abd2158cc1d1a74326184a9e8\": rpc error: code = NotFound desc = could not find container \"50a16d908515fb7b399b9dc6041ecb82c20cb78abd2158cc1d1a74326184a9e8\": container with ID starting with 50a16d908515fb7b399b9dc6041ecb82c20cb78abd2158cc1d1a74326184a9e8 not found: ID does not exist"
Jul 23 17:42:52.999377 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:42:52.999351    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-219aa1/test-model-219aa1-kserve-6b64757995-xnntf"]
Jul 23 17:42:53.003379 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:42:53.003357    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-219aa1/test-model-219aa1-kserve-6b64757995-xnntf"]
Jul 23 17:42:53.034367 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:42:53.034340    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba76055b-2a37-43eb-96f9-1d95c1edabd1-dshm\") on node \"ip-10-0-129-216.ec2.internal\" DevicePath \"\""
Jul 23 17:42:53.034367 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:42:53.034365    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba76055b-2a37-43eb-96f9-1d95c1edabd1-home\") on node \"ip-10-0-129-216.ec2.internal\" DevicePath \"\""
Jul 23 17:42:53.034513 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:42:53.034374    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba76055b-2a37-43eb-96f9-1d95c1edabd1-model-cache\") on node \"ip-10-0-129-216.ec2.internal\" DevicePath \"\""
Jul 23 17:42:53.034513 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:42:53.034384    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba76055b-2a37-43eb-96f9-1d95c1edabd1-tls-certs\") on node \"ip-10-0-129-216.ec2.internal\" DevicePath \"\""
Jul 23 17:42:53.034513 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:42:53.034394    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sjgtr\" (UniqueName: \"kubernetes.io/projected/ba76055b-2a37-43eb-96f9-1d95c1edabd1-kube-api-access-sjgtr\") on node \"ip-10-0-129-216.ec2.internal\" DevicePath \"\""
Jul 23 17:42:53.034513 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:42:53.034403    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba76055b-2a37-43eb-96f9-1d95c1edabd1-tmp-dir\") on node \"ip-10-0-129-216.ec2.internal\" DevicePath \"\""
Jul 23 17:42:53.941547 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:42:53.941513    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:42:54.343124 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:42:54.343034    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:42:54.768190 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:42:54.768146    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ba76055b-2a37-43eb-96f9-1d95c1edabd1" path="/var/lib/kubelet/pods/ba76055b-2a37-43eb-96f9-1d95c1edabd1/volumes"
Jul 23 17:43:02.036589 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:43:02.036553    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:43:02.141084 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:43:02.141034    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:43:02.327021 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:43:02.326912    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:43:02.563388 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:43:02.563343    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:43:02.686737 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:43:02.686655    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7363d9-7p2pc"]
Jul 23 17:43:02.687357 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:43:02.687336    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ba76055b-2a37-43eb-96f9-1d95c1edabd1" containerName="main"
Jul 23 17:43:02.687436 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:43:02.687360    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="ba76055b-2a37-43eb-96f9-1d95c1edabd1" containerName="main"
Jul 23 17:43:02.687436 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:43:02.687394    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eeff7185-99fc-4943-b860-5de616062fad" containerName="maas-api"
Jul 23 17:43:02.687436 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:43:02.687404    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="eeff7185-99fc-4943-b860-5de616062fad" containerName="maas-api"
Jul 23 17:43:02.687538 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:43:02.687501    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="ba76055b-2a37-43eb-96f9-1d95c1edabd1" containerName="main"
Jul 23 17:43:02.687538 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:43:02.687515    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="eeff7185-99fc-4943-b860-5de616062fad" containerName="maas-api"
Jul 23 17:43:02.691016 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:43:02.691001    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7363d9-7p2pc"
Jul 23 17:43:02.694522 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:43:02.694499    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7wfmw\""
Jul 23 17:43:02.702814 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:43:02.702790    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7363d9-7p2pc"]
Jul 23 17:43:02.820056 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:43:02.820019    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/234f2eca-ffa9-44de-b290-6cf2b79c1aa0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-7363d9-7p2pc\" (UID: \"234f2eca-ffa9-44de-b290-6cf2b79c1aa0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7363d9-7p2pc"
Jul 23 17:43:02.920660 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:43:02.920629    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/234f2eca-ffa9-44de-b290-6cf2b79c1aa0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-7363d9-7p2pc\" (UID: \"234f2eca-ffa9-44de-b290-6cf2b79c1aa0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7363d9-7p2pc"
Jul 23 17:43:02.921237 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:43:02.921217    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/234f2eca-ffa9-44de-b290-6cf2b79c1aa0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-7363d9-7p2pc\" (UID: \"234f2eca-ffa9-44de-b290-6cf2b79c1aa0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7363d9-7p2pc"
Jul 23 17:43:03.002770 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:43:03.002739    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7363d9-7p2pc"
Jul 23 17:43:03.140751 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:43:03.140728    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7363d9-7p2pc"]
Jul 23 17:43:03.142837 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:43:03.142803    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod234f2eca_ffa9_44de_b290_6cf2b79c1aa0.slice/crio-538611eb1d70170d3cff0f43f7cbf7823c84a62d79fda2839986586841e964f5 WatchSource:0}: Error finding container 538611eb1d70170d3cff0f43f7cbf7823c84a62d79fda2839986586841e964f5: Status 404 returned error can't find the container with id 538611eb1d70170d3cff0f43f7cbf7823c84a62d79fda2839986586841e964f5
Jul 23 17:43:03.493820 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:43:03.493783    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:43:03.593641 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:43:03.593549    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:43:04.038486 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:43:04.038449    2577 generic.go:356] "Generic (PLEG): container finished" podID="234f2eca-ffa9-44de-b290-6cf2b79c1aa0" containerID="32ce7369805e742fbccb616125b51558c1719490356a3be3c7c3171375cdf42a" exitCode=0
Jul 23 17:43:04.038668 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:43:04.038536    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7363d9-7p2pc" event={"ID":"234f2eca-ffa9-44de-b290-6cf2b79c1aa0","Type":"ContainerDied","Data":"32ce7369805e742fbccb616125b51558c1719490356a3be3c7c3171375cdf42a"}
Jul 23 17:43:04.038668 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:43:04.038585    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7363d9-7p2pc" event={"ID":"234f2eca-ffa9-44de-b290-6cf2b79c1aa0","Type":"ContainerStarted","Data":"538611eb1d70170d3cff0f43f7cbf7823c84a62d79fda2839986586841e964f5"}
Jul 23 17:43:05.181195 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:43:05.181173    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7363d9-7p2pc"
Jul 23 17:43:05.243666 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:43:05.243597    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/234f2eca-ffa9-44de-b290-6cf2b79c1aa0-maas-api-service-ca\") pod \"234f2eca-ffa9-44de-b290-6cf2b79c1aa0\" (UID: \"234f2eca-ffa9-44de-b290-6cf2b79c1aa0\") "
Jul 23 17:43:05.243930 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:43:05.243908    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/234f2eca-ffa9-44de-b290-6cf2b79c1aa0-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "234f2eca-ffa9-44de-b290-6cf2b79c1aa0" (UID: "234f2eca-ffa9-44de-b290-6cf2b79c1aa0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 17:43:05.344196 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:43:05.344162    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/234f2eca-ffa9-44de-b290-6cf2b79c1aa0-maas-api-service-ca\") on node \"ip-10-0-129-216.ec2.internal\" DevicePath \"\""
Jul 23 17:43:06.051802 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:43:06.051776    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7363d9-7p2pc"
Jul 23 17:43:06.051979 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:43:06.051775    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7363d9-7p2pc" event={"ID":"234f2eca-ffa9-44de-b290-6cf2b79c1aa0","Type":"ContainerDied","Data":"538611eb1d70170d3cff0f43f7cbf7823c84a62d79fda2839986586841e964f5"}
Jul 23 17:43:06.051979 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:43:06.051899    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="538611eb1d70170d3cff0f43f7cbf7823c84a62d79fda2839986586841e964f5"
Jul 23 17:43:13.156402 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:43:13.156371    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-7363d9-7cc4448956-7h447"]
Jul 23 17:43:13.156867 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:43:13.156587    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7363d9-7cc4448956-7h447" podUID="f79ebd27-484a-4f75-915e-7ef4ba952817" containerName="maas-api" containerID="cri-o://676b601a59919ef432139db31bf4c5a4082f111f0793d998fa89f3c54e965800" gracePeriod=30
Jul 23 17:43:13.190601 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:43:13.190571    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:43:13.404478 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:43:13.404453    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7363d9-7cc4448956-7h447"
Jul 23 17:43:13.515619 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:43:13.515580    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f79ebd27-484a-4f75-915e-7ef4ba952817-openshift-service-ca\") pod \"f79ebd27-484a-4f75-915e-7ef4ba952817\" (UID: \"f79ebd27-484a-4f75-915e-7ef4ba952817\") "
Jul 23 17:43:13.515619 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:43:13.515634    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f79ebd27-484a-4f75-915e-7ef4ba952817-maas-api-tls\") pod \"f79ebd27-484a-4f75-915e-7ef4ba952817\" (UID: \"f79ebd27-484a-4f75-915e-7ef4ba952817\") "
Jul 23 17:43:13.515904 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:43:13.515760    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5k6wn\" (UniqueName: \"kubernetes.io/projected/f79ebd27-484a-4f75-915e-7ef4ba952817-kube-api-access-5k6wn\") pod \"f79ebd27-484a-4f75-915e-7ef4ba952817\" (UID: \"f79ebd27-484a-4f75-915e-7ef4ba952817\") "
Jul 23 17:43:13.516002 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:43:13.515970    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f79ebd27-484a-4f75-915e-7ef4ba952817-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "f79ebd27-484a-4f75-915e-7ef4ba952817" (UID: "f79ebd27-484a-4f75-915e-7ef4ba952817"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 17:43:13.517763 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:43:13.517729    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f79ebd27-484a-4f75-915e-7ef4ba952817-kube-api-access-5k6wn" (OuterVolumeSpecName: "kube-api-access-5k6wn") pod "f79ebd27-484a-4f75-915e-7ef4ba952817" (UID: "f79ebd27-484a-4f75-915e-7ef4ba952817"). InnerVolumeSpecName "kube-api-access-5k6wn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 17:43:13.517880 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:43:13.517815    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f79ebd27-484a-4f75-915e-7ef4ba952817-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "f79ebd27-484a-4f75-915e-7ef4ba952817" (UID: "f79ebd27-484a-4f75-915e-7ef4ba952817"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 17:43:13.616718 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:43:13.616685    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5k6wn\" (UniqueName: \"kubernetes.io/projected/f79ebd27-484a-4f75-915e-7ef4ba952817-kube-api-access-5k6wn\") on node \"ip-10-0-129-216.ec2.internal\" DevicePath \"\""
Jul 23 17:43:13.616718 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:43:13.616711    2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f79ebd27-484a-4f75-915e-7ef4ba952817-openshift-service-ca\") on node \"ip-10-0-129-216.ec2.internal\" DevicePath \"\""
Jul 23 17:43:13.616718 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:43:13.616722    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f79ebd27-484a-4f75-915e-7ef4ba952817-maas-api-tls\") on node \"ip-10-0-129-216.ec2.internal\" DevicePath \"\""
Jul 23 17:43:14.097675 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:43:14.097643    2577 generic.go:356] "Generic (PLEG): container finished" podID="f79ebd27-484a-4f75-915e-7ef4ba952817" containerID="676b601a59919ef432139db31bf4c5a4082f111f0793d998fa89f3c54e965800" exitCode=0
Jul 23 17:43:14.097859 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:43:14.097692    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7363d9-7cc4448956-7h447"
Jul 23 17:43:14.097859 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:43:14.097730    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7363d9-7cc4448956-7h447" event={"ID":"f79ebd27-484a-4f75-915e-7ef4ba952817","Type":"ContainerDied","Data":"676b601a59919ef432139db31bf4c5a4082f111f0793d998fa89f3c54e965800"}
Jul 23 17:43:14.097859 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:43:14.097787    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7363d9-7cc4448956-7h447" event={"ID":"f79ebd27-484a-4f75-915e-7ef4ba952817","Type":"ContainerDied","Data":"7e0423be5503ac88fc20c8831eafed50dcfeb208329b75b7483e899b51334953"}
Jul 23 17:43:14.097859 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:43:14.097815    2577 scope.go:117] "RemoveContainer" containerID="676b601a59919ef432139db31bf4c5a4082f111f0793d998fa89f3c54e965800"
Jul 23 17:43:14.108171 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:43:14.108154    2577 scope.go:117] "RemoveContainer" containerID="676b601a59919ef432139db31bf4c5a4082f111f0793d998fa89f3c54e965800"
Jul 23 17:43:14.108481 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:43:14.108462    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"676b601a59919ef432139db31bf4c5a4082f111f0793d998fa89f3c54e965800\": container with ID starting with 676b601a59919ef432139db31bf4c5a4082f111f0793d998fa89f3c54e965800 not found: ID does not exist" containerID="676b601a59919ef432139db31bf4c5a4082f111f0793d998fa89f3c54e965800"
Jul 23 17:43:14.108570 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:43:14.108493    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"676b601a59919ef432139db31bf4c5a4082f111f0793d998fa89f3c54e965800"} err="failed to get container status \"676b601a59919ef432139db31bf4c5a4082f111f0793d998fa89f3c54e965800\": rpc error: code = NotFound desc = could not find container \"676b601a59919ef432139db31bf4c5a4082f111f0793d998fa89f3c54e965800\": container with ID starting with 676b601a59919ef432139db31bf4c5a4082f111f0793d998fa89f3c54e965800 not found: ID does not exist"
Jul 23 17:43:14.120751 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:43:14.120729    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-7363d9-7cc4448956-7h447"]
Jul 23 17:43:14.126883 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:43:14.126862    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-7363d9-7cc4448956-7h447"]
Jul 23 17:43:14.228907 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:43:14.228866    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:43:14.765949 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:43:14.765907    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f79ebd27-484a-4f75-915e-7ef4ba952817" path="/var/lib/kubelet/pods/f79ebd27-484a-4f75-915e-7ef4ba952817/volumes"
Jul 23 17:43:17.808118 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:43:17.808081    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:43:17.819833 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:43:17.819800    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7363d9-7p2pc"]
Jul 23 17:43:17.823068 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:43:17.823037    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7363d9-7p2pc"]
Jul 23 17:43:18.264366 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:43:18.264333    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:43:18.767867 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:43:18.767830    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="234f2eca-ffa9-44de-b290-6cf2b79c1aa0" path="/var/lib/kubelet/pods/234f2eca-ffa9-44de-b290-6cf2b79c1aa0/volumes"
Jul 23 17:43:24.596085 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:43:24.596042    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:43:24.768755 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:43:24.768717    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:43:24.922297 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:43:24.922199    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:43:25.137847 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:43:25.137817    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:43:25.524543 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:43:25.524509    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:43:26.040211 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:43:26.040171    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:43:26.203274 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:43:26.203229    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:43:28.790849 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:43:28.790817    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:43:28.903696 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:43:28.903649    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:43:28.906744 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:43:28.906709    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f9b147-584676645c-cbk9z"]
Jul 23 17:43:28.907123 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:43:28.907110    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="234f2eca-ffa9-44de-b290-6cf2b79c1aa0" containerName="revoke-keys"
Jul 23 17:43:28.907123 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:43:28.907123    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="234f2eca-ffa9-44de-b290-6cf2b79c1aa0" containerName="revoke-keys"
Jul 23 17:43:28.907218 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:43:28.907145    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f79ebd27-484a-4f75-915e-7ef4ba952817" containerName="maas-api"
Jul 23 17:43:28.907218 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:43:28.907151    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="f79ebd27-484a-4f75-915e-7ef4ba952817" containerName="maas-api"
Jul 23 17:43:28.907218 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:43:28.907205    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="234f2eca-ffa9-44de-b290-6cf2b79c1aa0" containerName="revoke-keys"
Jul 23 17:43:28.907218 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:43:28.907216    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="f79ebd27-484a-4f75-915e-7ef4ba952817" containerName="maas-api"
Jul 23 17:43:28.913294 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:43:28.913272    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f9b147-584676645c-cbk9z"
Jul 23 17:43:28.919875 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:43:28.919845    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-f9b147-serving-cert\""
Jul 23 17:43:28.931546 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:43:28.931517    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f9b147-584676645c-cbk9z"]
Jul 23 17:43:28.960616 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:43:28.960587    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/af85aba0-b2e2-4c0b-9df7-323e4f80e3d0-maas-api-tls\") pod \"maas-api-e2e-api-a-f9b147-584676645c-cbk9z\" (UID: \"af85aba0-b2e2-4c0b-9df7-323e4f80e3d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f9b147-584676645c-cbk9z"
Jul 23 17:43:28.960742 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:43:28.960635    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/af85aba0-b2e2-4c0b-9df7-323e4f80e3d0-openshift-service-ca\") pod \"maas-api-e2e-api-a-f9b147-584676645c-cbk9z\" (UID: \"af85aba0-b2e2-4c0b-9df7-323e4f80e3d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f9b147-584676645c-cbk9z"
Jul 23 17:43:28.960742 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:43:28.960664    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kb9qs\" (UniqueName: \"kubernetes.io/projected/af85aba0-b2e2-4c0b-9df7-323e4f80e3d0-kube-api-access-kb9qs\") pod \"maas-api-e2e-api-a-f9b147-584676645c-cbk9z\" (UID: \"af85aba0-b2e2-4c0b-9df7-323e4f80e3d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f9b147-584676645c-cbk9z"
Jul 23 17:43:29.035664 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:43:29.035636    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:43:29.061287 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:43:29.061200    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/af85aba0-b2e2-4c0b-9df7-323e4f80e3d0-openshift-service-ca\") pod \"maas-api-e2e-api-a-f9b147-584676645c-cbk9z\" (UID: \"af85aba0-b2e2-4c0b-9df7-323e4f80e3d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f9b147-584676645c-cbk9z"
Jul 23 17:43:29.061287 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:43:29.061239    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kb9qs\" (UniqueName: \"kubernetes.io/projected/af85aba0-b2e2-4c0b-9df7-323e4f80e3d0-kube-api-access-kb9qs\") pod \"maas-api-e2e-api-a-f9b147-584676645c-cbk9z\" (UID: \"af85aba0-b2e2-4c0b-9df7-323e4f80e3d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f9b147-584676645c-cbk9z"
Jul 23 17:43:29.061491 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:43:29.061357    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/af85aba0-b2e2-4c0b-9df7-323e4f80e3d0-maas-api-tls\") pod \"maas-api-e2e-api-a-f9b147-584676645c-cbk9z\" (UID: \"af85aba0-b2e2-4c0b-9df7-323e4f80e3d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f9b147-584676645c-cbk9z"
Jul 23 17:43:29.061806 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:43:29.061787    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/af85aba0-b2e2-4c0b-9df7-323e4f80e3d0-openshift-service-ca\") pod \"maas-api-e2e-api-a-f9b147-584676645c-cbk9z\" (UID: \"af85aba0-b2e2-4c0b-9df7-323e4f80e3d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f9b147-584676645c-cbk9z"
Jul 23 17:43:29.063887 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:43:29.063861    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/af85aba0-b2e2-4c0b-9df7-323e4f80e3d0-maas-api-tls\") pod \"maas-api-e2e-api-a-f9b147-584676645c-cbk9z\" (UID: \"af85aba0-b2e2-4c0b-9df7-323e4f80e3d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f9b147-584676645c-cbk9z"
Jul 23 17:43:29.074285 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:43:29.074228    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kb9qs\" (UniqueName: \"kubernetes.io/projected/af85aba0-b2e2-4c0b-9df7-323e4f80e3d0-kube-api-access-kb9qs\") pod \"maas-api-e2e-api-a-f9b147-584676645c-cbk9z\" (UID: \"af85aba0-b2e2-4c0b-9df7-323e4f80e3d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f9b147-584676645c-cbk9z"
Jul 23 17:43:29.208194 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:43:29.208158    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:43:29.224694 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:43:29.224666    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f9b147-584676645c-cbk9z"
Jul 23 17:43:29.378650 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:43:29.378625    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f9b147-584676645c-cbk9z"]
Jul 23 17:43:29.379311 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:43:29.379283    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf85aba0_b2e2_4c0b_9df7_323e4f80e3d0.slice/crio-a64654953e42deafaee6de5d361dffd0fb2837cd18aca9183332328276f987bf WatchSource:0}: Error finding container a64654953e42deafaee6de5d361dffd0fb2837cd18aca9183332328276f987bf: Status 404 returned error can't find the container with id a64654953e42deafaee6de5d361dffd0fb2837cd18aca9183332328276f987bf
Jul 23 17:43:29.957670 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:43:29.957631    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:43:30.103407 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:43:30.103373    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:43:30.197276 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:43:30.197226    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f9b147-584676645c-cbk9z" event={"ID":"af85aba0-b2e2-4c0b-9df7-323e4f80e3d0","Type":"ContainerStarted","Data":"a64654953e42deafaee6de5d361dffd0fb2837cd18aca9183332328276f987bf"}
Jul 23 17:43:31.204518 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:43:31.204480    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f9b147-584676645c-cbk9z" event={"ID":"af85aba0-b2e2-4c0b-9df7-323e4f80e3d0","Type":"ContainerStarted","Data":"83cda186d47f3036da9bbee134782c6a810ebcd42fd011130b290468bdd392b7"}
Jul 23 17:43:31.204914 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:43:31.204606    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f9b147-584676645c-cbk9z"
Jul 23 17:43:31.236538 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:43:31.236492    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f9b147-584676645c-cbk9z" podStartSLOduration=2.122744627 podStartE2EDuration="3.236475774s" totalImagesPullingTime="1.113731147s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 17:43:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 17:43:31.233701532 +0000 UTC m=+2307.003987053" watchObservedRunningTime="2026-07-23 17:43:31.236475774 +0000 UTC m=+2307.006761293"
Jul 23 17:43:33.383795 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:43:33.383766    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:43:34.020208 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:43:34.020169    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:43:34.415147 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:43:34.415065    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:43:35.101164 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:43:35.101128    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:43:37.215984 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:43:37.215952    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f9b147-584676645c-cbk9z"
Jul 23 17:43:37.920314 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:43:37.920280    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:43:38.031451 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:43:38.031415    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-6f74dd-68f9659b8d-d29wr"]
Jul 23 17:43:38.036680 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:43:38.036650    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6f74dd-68f9659b8d-d29wr"
Jul 23 17:43:38.041677 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:43:38.041385    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-6f74dd-serving-cert\""
Jul 23 17:43:38.074566 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:43:38.074536    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-6f74dd-68f9659b8d-d29wr"]
Jul 23 17:43:38.138444 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:43:38.138405    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/245eb040-a556-416b-8f43-98e2ee471c51-openshift-service-ca\") pod \"maas-api-e2e-api-b-6f74dd-68f9659b8d-d29wr\" (UID: \"245eb040-a556-416b-8f43-98e2ee471c51\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6f74dd-68f9659b8d-d29wr"
Jul 23 17:43:38.138615 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:43:38.138470    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lnz88\" (UniqueName: \"kubernetes.io/projected/245eb040-a556-416b-8f43-98e2ee471c51-kube-api-access-lnz88\") pod \"maas-api-e2e-api-b-6f74dd-68f9659b8d-d29wr\" (UID: \"245eb040-a556-416b-8f43-98e2ee471c51\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6f74dd-68f9659b8d-d29wr"
Jul 23 17:43:38.138615 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:43:38.138527    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/245eb040-a556-416b-8f43-98e2ee471c51-maas-api-tls\") pod \"maas-api-e2e-api-b-6f74dd-68f9659b8d-d29wr\" (UID: \"245eb040-a556-416b-8f43-98e2ee471c51\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6f74dd-68f9659b8d-d29wr"
Jul 23 17:43:38.239284 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:43:38.239238    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/245eb040-a556-416b-8f43-98e2ee471c51-openshift-service-ca\") pod \"maas-api-e2e-api-b-6f74dd-68f9659b8d-d29wr\" (UID: \"245eb040-a556-416b-8f43-98e2ee471c51\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6f74dd-68f9659b8d-d29wr"
Jul 23 17:43:38.239663 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:43:38.239314    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lnz88\" (UniqueName: \"kubernetes.io/projected/245eb040-a556-416b-8f43-98e2ee471c51-kube-api-access-lnz88\") pod \"maas-api-e2e-api-b-6f74dd-68f9659b8d-d29wr\" (UID: \"245eb040-a556-416b-8f43-98e2ee471c51\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6f74dd-68f9659b8d-d29wr"
Jul 23 17:43:38.239663 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:43:38.239374    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/245eb040-a556-416b-8f43-98e2ee471c51-maas-api-tls\") pod \"maas-api-e2e-api-b-6f74dd-68f9659b8d-d29wr\" (UID: \"245eb040-a556-416b-8f43-98e2ee471c51\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6f74dd-68f9659b8d-d29wr"
Jul 23 17:43:38.239963 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:43:38.239943    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/245eb040-a556-416b-8f43-98e2ee471c51-openshift-service-ca\") pod \"maas-api-e2e-api-b-6f74dd-68f9659b8d-d29wr\" (UID: \"245eb040-a556-416b-8f43-98e2ee471c51\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6f74dd-68f9659b8d-d29wr"
Jul 23 17:43:38.241743 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:43:38.241724    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/245eb040-a556-416b-8f43-98e2ee471c51-maas-api-tls\") pod \"maas-api-e2e-api-b-6f74dd-68f9659b8d-d29wr\" (UID: \"245eb040-a556-416b-8f43-98e2ee471c51\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6f74dd-68f9659b8d-d29wr"
Jul 23 17:43:38.248642 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:43:38.248620    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lnz88\" (UniqueName: \"kubernetes.io/projected/245eb040-a556-416b-8f43-98e2ee471c51-kube-api-access-lnz88\") pod \"maas-api-e2e-api-b-6f74dd-68f9659b8d-d29wr\" (UID: \"245eb040-a556-416b-8f43-98e2ee471c51\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6f74dd-68f9659b8d-d29wr"
Jul 23 17:43:38.350972 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:43:38.350943    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6f74dd-68f9659b8d-d29wr"
Jul 23 17:43:38.508767 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:43:38.508740    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-6f74dd-68f9659b8d-d29wr"]
Jul 23 17:43:38.509532 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:43:38.509508    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod245eb040_a556_416b_8f43_98e2ee471c51.slice/crio-b368214a93df9de257eb30597beb9e58501853564742149b080d84f06754728c WatchSource:0}: Error finding container b368214a93df9de257eb30597beb9e58501853564742149b080d84f06754728c: Status 404 returned error can't find the container with id b368214a93df9de257eb30597beb9e58501853564742149b080d84f06754728c
Jul 23 17:43:38.510746 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:43:38.510730    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 23 17:43:38.583123 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:43:38.583084    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:43:38.724706 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:43:38.724676    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:43:38.822970 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:43:38.822896    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:43:38.917766 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:43:38.917734    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:43:39.259313 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:43:39.259279    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6f74dd-68f9659b8d-d29wr" event={"ID":"245eb040-a556-416b-8f43-98e2ee471c51","Type":"ContainerStarted","Data":"b368214a93df9de257eb30597beb9e58501853564742149b080d84f06754728c"}
Jul 23 17:43:40.268348 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:43:40.268310    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6f74dd-68f9659b8d-d29wr" event={"ID":"245eb040-a556-416b-8f43-98e2ee471c51","Type":"ContainerStarted","Data":"a7c45f7e4256eff3abee8d9c82069b881720b68924848b677bce98cd5a61d568"}
Jul 23 17:43:40.268724 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:43:40.268412    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6f74dd-68f9659b8d-d29wr"
Jul 23 17:43:40.309564 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:43:40.309513    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6f74dd-68f9659b8d-d29wr" podStartSLOduration=1.8869833040000001 podStartE2EDuration="3.309493453s" totalImagesPullingTime="1.422510149s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 17:43:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 17:43:40.308116635 +0000 UTC m=+2316.078402203" watchObservedRunningTime="2026-07-23 17:43:40.309493453 +0000 UTC m=+2316.079778974"
Jul 23 17:43:42.422542 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:43:42.422496    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:43:46.279370 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:43:46.279339    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6f74dd-68f9659b8d-d29wr"
Jul 23 17:43:50.790608 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:43:50.790569    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f9b147-zrfwk"]
Jul 23 17:43:50.794594 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:43:50.794569    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f9b147-zrfwk"
Jul 23 17:43:50.796951 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:43:50.796930    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7wfmw\""
Jul 23 17:43:50.803617 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:43:50.803595    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f9b147-zrfwk"]
Jul 23 17:43:50.850386 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:43:50.850338    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e17de8fa-4fdc-442d-ad75-cb60ac7c9709-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-f9b147-zrfwk\" (UID: \"e17de8fa-4fdc-442d-ad75-cb60ac7c9709\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f9b147-zrfwk"
Jul 23 17:43:50.950967 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:43:50.950930    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e17de8fa-4fdc-442d-ad75-cb60ac7c9709-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-f9b147-zrfwk\" (UID: \"e17de8fa-4fdc-442d-ad75-cb60ac7c9709\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f9b147-zrfwk"
Jul 23 17:43:50.951622 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:43:50.951603    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e17de8fa-4fdc-442d-ad75-cb60ac7c9709-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-f9b147-zrfwk\" (UID: \"e17de8fa-4fdc-442d-ad75-cb60ac7c9709\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f9b147-zrfwk"
Jul 23 17:43:51.106492 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:43:51.106414    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f9b147-zrfwk"
Jul 23 17:43:51.172218 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:43:51.172148    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:43:51.256597 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:43:51.256567    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f9b147-zrfwk"]
Jul 23 17:43:51.257937 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:43:51.257907    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode17de8fa_4fdc_442d_ad75_cb60ac7c9709.slice/crio-a192d5e2e453cd086af43eb66ddba4eff5fa6b990a4dc898e157efedd2842c83 WatchSource:0}: Error finding container a192d5e2e453cd086af43eb66ddba4eff5fa6b990a4dc898e157efedd2842c83: Status 404 returned error can't find the container with id a192d5e2e453cd086af43eb66ddba4eff5fa6b990a4dc898e157efedd2842c83
Jul 23 17:43:51.294233 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:43:51.294206    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:43:51.330174 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:43:51.330138    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f9b147-zrfwk" event={"ID":"e17de8fa-4fdc-442d-ad75-cb60ac7c9709","Type":"ContainerStarted","Data":"a192d5e2e453cd086af43eb66ddba4eff5fa6b990a4dc898e157efedd2842c83"}
Jul 23 17:43:52.337340 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:43:52.337303    2577 generic.go:356] "Generic (PLEG): container finished" podID="e17de8fa-4fdc-442d-ad75-cb60ac7c9709" containerID="f09e2f0a3323914316b19f80f8920c77d7b7ac3ab36541f85ef23d0a22d73ea3" exitCode=0
Jul 23 17:43:52.337720 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:43:52.337361    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f9b147-zrfwk" event={"ID":"e17de8fa-4fdc-442d-ad75-cb60ac7c9709","Type":"ContainerDied","Data":"f09e2f0a3323914316b19f80f8920c77d7b7ac3ab36541f85ef23d0a22d73ea3"}
Jul 23 17:43:53.480718 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:43:53.480694    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f9b147-zrfwk"
Jul 23 17:43:53.576209 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:43:53.576180    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e17de8fa-4fdc-442d-ad75-cb60ac7c9709-maas-api-service-ca\") pod \"e17de8fa-4fdc-442d-ad75-cb60ac7c9709\" (UID: \"e17de8fa-4fdc-442d-ad75-cb60ac7c9709\") "
Jul 23 17:43:53.576530 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:43:53.576506    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e17de8fa-4fdc-442d-ad75-cb60ac7c9709-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e17de8fa-4fdc-442d-ad75-cb60ac7c9709" (UID: "e17de8fa-4fdc-442d-ad75-cb60ac7c9709"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 17:43:53.677632 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:43:53.677555    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e17de8fa-4fdc-442d-ad75-cb60ac7c9709-maas-api-service-ca\") on node \"ip-10-0-129-216.ec2.internal\" DevicePath \"\""
Jul 23 17:43:54.350994 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:43:54.350962    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f9b147-zrfwk"
Jul 23 17:43:54.351174 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:43:54.351008    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f9b147-zrfwk" event={"ID":"e17de8fa-4fdc-442d-ad75-cb60ac7c9709","Type":"ContainerDied","Data":"a192d5e2e453cd086af43eb66ddba4eff5fa6b990a4dc898e157efedd2842c83"}
Jul 23 17:43:54.351174 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:43:54.351054    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a192d5e2e453cd086af43eb66ddba4eff5fa6b990a4dc898e157efedd2842c83"
Jul 23 17:44:03.628385 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:44:03.628298    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:44:03.728460 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:44:03.728428    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:44:04.508962 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:44:04.508926    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:44:04.990698 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:44:04.990665    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:44:05.001020 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:44:05.000993    2577 scope.go:117] "RemoveContainer" containerID="ccafc5abd6e601a13657e4f07b7ac3d71ceef30aa2953870ea833e827ab07901"
Jul 23 17:44:06.396527 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:44:06.396496    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f9b147-584676645c-cbk9z"]
Jul 23 17:44:06.397033 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:44:06.396722    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f9b147-584676645c-cbk9z" podUID="af85aba0-b2e2-4c0b-9df7-323e4f80e3d0" containerName="maas-api" containerID="cri-o://83cda186d47f3036da9bbee134782c6a810ebcd42fd011130b290468bdd392b7" gracePeriod=30
Jul 23 17:44:06.430271 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:44:06.430227    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:44:06.646687 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:44:06.646618    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f9b147-584676645c-cbk9z"
Jul 23 17:44:06.790562 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:44:06.790534    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/af85aba0-b2e2-4c0b-9df7-323e4f80e3d0-openshift-service-ca\") pod \"af85aba0-b2e2-4c0b-9df7-323e4f80e3d0\" (UID: \"af85aba0-b2e2-4c0b-9df7-323e4f80e3d0\") "
Jul 23 17:44:06.790748 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:44:06.790604    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/af85aba0-b2e2-4c0b-9df7-323e4f80e3d0-maas-api-tls\") pod \"af85aba0-b2e2-4c0b-9df7-323e4f80e3d0\" (UID: \"af85aba0-b2e2-4c0b-9df7-323e4f80e3d0\") "
Jul 23 17:44:06.790823 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:44:06.790763    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kb9qs\" (UniqueName: \"kubernetes.io/projected/af85aba0-b2e2-4c0b-9df7-323e4f80e3d0-kube-api-access-kb9qs\") pod \"af85aba0-b2e2-4c0b-9df7-323e4f80e3d0\" (UID: \"af85aba0-b2e2-4c0b-9df7-323e4f80e3d0\") "
Jul 23 17:44:06.790943 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:44:06.790922    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af85aba0-b2e2-4c0b-9df7-323e4f80e3d0-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "af85aba0-b2e2-4c0b-9df7-323e4f80e3d0" (UID: "af85aba0-b2e2-4c0b-9df7-323e4f80e3d0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 17:44:06.791099 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:44:06.791084    2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/af85aba0-b2e2-4c0b-9df7-323e4f80e3d0-openshift-service-ca\") on node \"ip-10-0-129-216.ec2.internal\" DevicePath \"\""
Jul 23 17:44:06.792638 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:44:06.792612    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af85aba0-b2e2-4c0b-9df7-323e4f80e3d0-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "af85aba0-b2e2-4c0b-9df7-323e4f80e3d0" (UID: "af85aba0-b2e2-4c0b-9df7-323e4f80e3d0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 17:44:06.792768 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:44:06.792751    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af85aba0-b2e2-4c0b-9df7-323e4f80e3d0-kube-api-access-kb9qs" (OuterVolumeSpecName: "kube-api-access-kb9qs") pod "af85aba0-b2e2-4c0b-9df7-323e4f80e3d0" (UID: "af85aba0-b2e2-4c0b-9df7-323e4f80e3d0"). InnerVolumeSpecName "kube-api-access-kb9qs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 17:44:06.891735 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:44:06.891697    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/af85aba0-b2e2-4c0b-9df7-323e4f80e3d0-maas-api-tls\") on node \"ip-10-0-129-216.ec2.internal\" DevicePath \"\""
Jul 23 17:44:06.891735 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:44:06.891728    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kb9qs\" (UniqueName: \"kubernetes.io/projected/af85aba0-b2e2-4c0b-9df7-323e4f80e3d0-kube-api-access-kb9qs\") on node \"ip-10-0-129-216.ec2.internal\" DevicePath \"\""
Jul 23 17:44:07.425195 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:44:07.425162    2577 generic.go:356] "Generic (PLEG): container finished" podID="af85aba0-b2e2-4c0b-9df7-323e4f80e3d0" containerID="83cda186d47f3036da9bbee134782c6a810ebcd42fd011130b290468bdd392b7" exitCode=0
Jul 23 17:44:07.425652 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:44:07.425209    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f9b147-584676645c-cbk9z"
Jul 23 17:44:07.425652 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:44:07.425241    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f9b147-584676645c-cbk9z" event={"ID":"af85aba0-b2e2-4c0b-9df7-323e4f80e3d0","Type":"ContainerDied","Data":"83cda186d47f3036da9bbee134782c6a810ebcd42fd011130b290468bdd392b7"}
Jul 23 17:44:07.425652 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:44:07.425308    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f9b147-584676645c-cbk9z" event={"ID":"af85aba0-b2e2-4c0b-9df7-323e4f80e3d0","Type":"ContainerDied","Data":"a64654953e42deafaee6de5d361dffd0fb2837cd18aca9183332328276f987bf"}
Jul 23 17:44:07.425652 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:44:07.425329    2577 scope.go:117] "RemoveContainer" containerID="83cda186d47f3036da9bbee134782c6a810ebcd42fd011130b290468bdd392b7"
Jul 23 17:44:07.436583 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:44:07.436562    2577 scope.go:117] "RemoveContainer" containerID="83cda186d47f3036da9bbee134782c6a810ebcd42fd011130b290468bdd392b7"
Jul 23 17:44:07.436887 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:44:07.436863    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83cda186d47f3036da9bbee134782c6a810ebcd42fd011130b290468bdd392b7\": container with ID starting with 83cda186d47f3036da9bbee134782c6a810ebcd42fd011130b290468bdd392b7 not found: ID does not exist" containerID="83cda186d47f3036da9bbee134782c6a810ebcd42fd011130b290468bdd392b7"
Jul 23 17:44:07.436971 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:44:07.436898    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83cda186d47f3036da9bbee134782c6a810ebcd42fd011130b290468bdd392b7"} err="failed to get container status \"83cda186d47f3036da9bbee134782c6a810ebcd42fd011130b290468bdd392b7\": rpc error: code = NotFound desc = could not find container \"83cda186d47f3036da9bbee134782c6a810ebcd42fd011130b290468bdd392b7\": container with ID starting with 83cda186d47f3036da9bbee134782c6a810ebcd42fd011130b290468bdd392b7 not found: ID does not exist"
Jul 23 17:44:07.448085 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:44:07.448063    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f9b147-584676645c-cbk9z"]
Jul 23 17:44:07.454465 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:44:07.454443    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f9b147-584676645c-cbk9z"]
Jul 23 17:44:08.766425 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:44:08.766387    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="af85aba0-b2e2-4c0b-9df7-323e4f80e3d0" path="/var/lib/kubelet/pods/af85aba0-b2e2-4c0b-9df7-323e4f80e3d0/volumes"
Jul 23 17:44:10.878472 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:44:10.878432    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f9b147-zrfwk"]
Jul 23 17:44:10.881368 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:44:10.881342    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f9b147-zrfwk"]
Jul 23 17:44:10.881557 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:44:10.881533    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:44:11.288082 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:44:11.288051    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:44:12.765905 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:44:12.765872    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e17de8fa-4fdc-442d-ad75-cb60ac7c9709" path="/var/lib/kubelet/pods/e17de8fa-4fdc-442d-ad75-cb60ac7c9709/volumes"
Jul 23 17:44:17.692978 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:44:17.692945    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:44:17.768101 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:44:17.768060    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:44:18.226871 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:44:18.226842    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:44:18.509317 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:44:18.509222    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:44:18.649265 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:44:18.649212    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6f74dd-xcjxp"]
Jul 23 17:44:18.649889 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:44:18.649867    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e17de8fa-4fdc-442d-ad75-cb60ac7c9709" containerName="revoke-keys"
Jul 23 17:44:18.649955 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:44:18.649892    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="e17de8fa-4fdc-442d-ad75-cb60ac7c9709" containerName="revoke-keys"
Jul 23 17:44:18.649955 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:44:18.649907    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af85aba0-b2e2-4c0b-9df7-323e4f80e3d0" containerName="maas-api"
Jul 23 17:44:18.649955 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:44:18.649916    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="af85aba0-b2e2-4c0b-9df7-323e4f80e3d0" containerName="maas-api"
Jul 23 17:44:18.650077 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:44:18.650017    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="af85aba0-b2e2-4c0b-9df7-323e4f80e3d0" containerName="maas-api"
Jul 23 17:44:18.650077 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:44:18.650028    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="e17de8fa-4fdc-442d-ad75-cb60ac7c9709" containerName="revoke-keys"
Jul 23 17:44:18.653934 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:44:18.653912    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6f74dd-xcjxp"
Jul 23 17:44:18.656444 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:44:18.656418    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7wfmw\""
Jul 23 17:44:18.659124 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:44:18.659100    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6f74dd-xcjxp"]
Jul 23 17:44:18.797217 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:44:18.797130    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1014ac1-988e-47e8-b409-ececf480e8c0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-6f74dd-xcjxp\" (UID: \"f1014ac1-988e-47e8-b409-ececf480e8c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6f74dd-xcjxp"
Jul 23 17:44:18.898544 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:44:18.898513    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1014ac1-988e-47e8-b409-ececf480e8c0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-6f74dd-xcjxp\" (UID: \"f1014ac1-988e-47e8-b409-ececf480e8c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6f74dd-xcjxp"
Jul 23 17:44:18.899116 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:44:18.899093    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1014ac1-988e-47e8-b409-ececf480e8c0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-6f74dd-xcjxp\" (UID: \"f1014ac1-988e-47e8-b409-ececf480e8c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6f74dd-xcjxp"
Jul 23 17:44:18.967183 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:44:18.967151    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6f74dd-xcjxp"
Jul 23 17:44:19.107803 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:44:19.107760    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6f74dd-xcjxp"]
Jul 23 17:44:19.108544 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:44:19.108515    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1014ac1_988e_47e8_b409_ececf480e8c0.slice/crio-4b83cb44069b2f1bab43e5cce8d9422085c811a0401597856dff2486ba1e879e WatchSource:0}: Error finding container 4b83cb44069b2f1bab43e5cce8d9422085c811a0401597856dff2486ba1e879e: Status 404 returned error can't find the container with id 4b83cb44069b2f1bab43e5cce8d9422085c811a0401597856dff2486ba1e879e
Jul 23 17:44:19.501059 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:44:19.501017    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6f74dd-xcjxp" event={"ID":"f1014ac1-988e-47e8-b409-ececf480e8c0","Type":"ContainerStarted","Data":"a25d9be00a507c87ec52feff1be38ba8a665fa3bda88c34cb9743dc5691a49f0"}
Jul 23 17:44:19.501222 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:44:19.501078    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6f74dd-xcjxp" event={"ID":"f1014ac1-988e-47e8-b409-ececf480e8c0","Type":"ContainerStarted","Data":"4b83cb44069b2f1bab43e5cce8d9422085c811a0401597856dff2486ba1e879e"}
Jul 23 17:44:19.518721 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:44:19.518667    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6f74dd-xcjxp" podStartSLOduration=1.5186510439999998 podStartE2EDuration="1.518651044s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 17:44:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 17:44:19.518452709 +0000 UTC m=+2355.288738228" watchObservedRunningTime="2026-07-23 17:44:19.518651044 +0000 UTC m=+2355.288936563"
Jul 23 17:44:20.508652 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:44:20.508618    2577 generic.go:356] "Generic (PLEG): container finished" podID="f1014ac1-988e-47e8-b409-ececf480e8c0" containerID="a25d9be00a507c87ec52feff1be38ba8a665fa3bda88c34cb9743dc5691a49f0" exitCode=0
Jul 23 17:44:20.509040 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:44:20.508702    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6f74dd-xcjxp" event={"ID":"f1014ac1-988e-47e8-b409-ececf480e8c0","Type":"ContainerDied","Data":"a25d9be00a507c87ec52feff1be38ba8a665fa3bda88c34cb9743dc5691a49f0"}
Jul 23 17:44:21.672279 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:44:21.672240    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6f74dd-xcjxp"
Jul 23 17:44:21.746722 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:44:21.746691    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1014ac1-988e-47e8-b409-ececf480e8c0-maas-api-service-ca\") pod \"f1014ac1-988e-47e8-b409-ececf480e8c0\" (UID: \"f1014ac1-988e-47e8-b409-ececf480e8c0\") "
Jul 23 17:44:21.747068 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:44:21.747047    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1014ac1-988e-47e8-b409-ececf480e8c0-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f1014ac1-988e-47e8-b409-ececf480e8c0" (UID: "f1014ac1-988e-47e8-b409-ececf480e8c0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 17:44:21.848041 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:44:21.847958    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1014ac1-988e-47e8-b409-ececf480e8c0-maas-api-service-ca\") on node \"ip-10-0-129-216.ec2.internal\" DevicePath \"\""
Jul 23 17:44:22.521308 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:44:22.521280    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6f74dd-xcjxp"
Jul 23 17:44:22.521460 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:44:22.521283    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6f74dd-xcjxp" event={"ID":"f1014ac1-988e-47e8-b409-ececf480e8c0","Type":"ContainerDied","Data":"4b83cb44069b2f1bab43e5cce8d9422085c811a0401597856dff2486ba1e879e"}
Jul 23 17:44:22.521460 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:44:22.521404    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4b83cb44069b2f1bab43e5cce8d9422085c811a0401597856dff2486ba1e879e"
Jul 23 17:44:34.182677 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:44:34.182632    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-6f74dd-68f9659b8d-d29wr"]
Jul 23 17:44:34.184060 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:44:34.182945    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6f74dd-68f9659b8d-d29wr" podUID="245eb040-a556-416b-8f43-98e2ee471c51" containerName="maas-api" containerID="cri-o://a7c45f7e4256eff3abee8d9c82069b881720b68924848b677bce98cd5a61d568" gracePeriod=30
Jul 23 17:44:34.210906 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:44:34.210883    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:44:34.437947 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:44:34.437885    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6f74dd-68f9659b8d-d29wr"
Jul 23 17:44:34.559230 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:44:34.559200    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/245eb040-a556-416b-8f43-98e2ee471c51-maas-api-tls\") pod \"245eb040-a556-416b-8f43-98e2ee471c51\" (UID: \"245eb040-a556-416b-8f43-98e2ee471c51\") "
Jul 23 17:44:34.559440 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:44:34.559288    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/245eb040-a556-416b-8f43-98e2ee471c51-openshift-service-ca\") pod \"245eb040-a556-416b-8f43-98e2ee471c51\" (UID: \"245eb040-a556-416b-8f43-98e2ee471c51\") "
Jul 23 17:44:34.559440 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:44:34.559345    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lnz88\" (UniqueName: \"kubernetes.io/projected/245eb040-a556-416b-8f43-98e2ee471c51-kube-api-access-lnz88\") pod \"245eb040-a556-416b-8f43-98e2ee471c51\" (UID: \"245eb040-a556-416b-8f43-98e2ee471c51\") "
Jul 23 17:44:34.559641 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:44:34.559616    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/245eb040-a556-416b-8f43-98e2ee471c51-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "245eb040-a556-416b-8f43-98e2ee471c51" (UID: "245eb040-a556-416b-8f43-98e2ee471c51"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 17:44:34.561310 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:44:34.561283    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/245eb040-a556-416b-8f43-98e2ee471c51-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "245eb040-a556-416b-8f43-98e2ee471c51" (UID: "245eb040-a556-416b-8f43-98e2ee471c51"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 17:44:34.561466 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:44:34.561447    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/245eb040-a556-416b-8f43-98e2ee471c51-kube-api-access-lnz88" (OuterVolumeSpecName: "kube-api-access-lnz88") pod "245eb040-a556-416b-8f43-98e2ee471c51" (UID: "245eb040-a556-416b-8f43-98e2ee471c51"). InnerVolumeSpecName "kube-api-access-lnz88". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 17:44:34.591946 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:44:34.591914    2577 generic.go:356] "Generic (PLEG): container finished" podID="245eb040-a556-416b-8f43-98e2ee471c51" containerID="a7c45f7e4256eff3abee8d9c82069b881720b68924848b677bce98cd5a61d568" exitCode=0
Jul 23 17:44:34.592069 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:44:34.591954    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6f74dd-68f9659b8d-d29wr"
Jul 23 17:44:34.592069 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:44:34.591980    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6f74dd-68f9659b8d-d29wr" event={"ID":"245eb040-a556-416b-8f43-98e2ee471c51","Type":"ContainerDied","Data":"a7c45f7e4256eff3abee8d9c82069b881720b68924848b677bce98cd5a61d568"}
Jul 23 17:44:34.592069 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:44:34.592025    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6f74dd-68f9659b8d-d29wr" event={"ID":"245eb040-a556-416b-8f43-98e2ee471c51","Type":"ContainerDied","Data":"b368214a93df9de257eb30597beb9e58501853564742149b080d84f06754728c"}
Jul 23 17:44:34.592069 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:44:34.592047    2577 scope.go:117] "RemoveContainer" containerID="a7c45f7e4256eff3abee8d9c82069b881720b68924848b677bce98cd5a61d568"
Jul 23 17:44:34.601776 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:44:34.601759    2577 scope.go:117] "RemoveContainer" containerID="a7c45f7e4256eff3abee8d9c82069b881720b68924848b677bce98cd5a61d568"
Jul 23 17:44:34.602008 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:44:34.601985    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7c45f7e4256eff3abee8d9c82069b881720b68924848b677bce98cd5a61d568\": container with ID starting with a7c45f7e4256eff3abee8d9c82069b881720b68924848b677bce98cd5a61d568 not found: ID does not exist" containerID="a7c45f7e4256eff3abee8d9c82069b881720b68924848b677bce98cd5a61d568"
Jul 23 17:44:34.602081 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:44:34.602019    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7c45f7e4256eff3abee8d9c82069b881720b68924848b677bce98cd5a61d568"} err="failed to get container status \"a7c45f7e4256eff3abee8d9c82069b881720b68924848b677bce98cd5a61d568\": rpc error: code = NotFound desc = could not find container \"a7c45f7e4256eff3abee8d9c82069b881720b68924848b677bce98cd5a61d568\": container with ID starting with a7c45f7e4256eff3abee8d9c82069b881720b68924848b677bce98cd5a61d568 not found: ID does not exist"
Jul 23 17:44:34.614267 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:44:34.614224    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-6f74dd-68f9659b8d-d29wr"]
Jul 23 17:44:34.616522 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:44:34.616504    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-6f74dd-68f9659b8d-d29wr"]
Jul 23 17:44:34.660800 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:44:34.660768    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/245eb040-a556-416b-8f43-98e2ee471c51-maas-api-tls\") on node \"ip-10-0-129-216.ec2.internal\" DevicePath \"\""
Jul 23 17:44:34.660800 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:44:34.660796    2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/245eb040-a556-416b-8f43-98e2ee471c51-openshift-service-ca\") on node \"ip-10-0-129-216.ec2.internal\" DevicePath \"\""
Jul 23 17:44:34.660800 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:44:34.660805    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lnz88\" (UniqueName: \"kubernetes.io/projected/245eb040-a556-416b-8f43-98e2ee471c51-kube-api-access-lnz88\") on node \"ip-10-0-129-216.ec2.internal\" DevicePath \"\""
Jul 23 17:44:34.766325 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:44:34.766293    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="245eb040-a556-416b-8f43-98e2ee471c51" path="/var/lib/kubelet/pods/245eb040-a556-416b-8f43-98e2ee471c51/volumes"
Jul 23 17:44:38.742668 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:44:38.742627    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:44:38.743440 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:44:38.743419    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6f74dd-xcjxp"]
Jul 23 17:44:38.748509 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:44:38.748483    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6f74dd-xcjxp"]
Jul 23 17:44:38.766249 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:44:38.766222    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f1014ac1-988e-47e8-b409-ececf480e8c0" path="/var/lib/kubelet/pods/f1014ac1-988e-47e8-b409-ececf480e8c0/volumes"
Jul 23 17:44:39.144167 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:44:39.144088    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:44:45.172278 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:44:45.172219    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:44:45.316846 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:44:45.316811    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:44:45.962023 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:44:45.961984    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:44:46.063083 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:44:46.063048    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:44:46.239798 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:44:46.239710    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:44:46.496169 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:44:46.496078    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:44:46.845675 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:44:46.845590    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:44:47.267372 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:44:47.267342    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:44:47.362773 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:44:47.362739    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:44:50.340869 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:44:50.340831    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:44:50.494877 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:44:50.494843    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-b9d590-ffdc5bcb6-qggkj"]
Jul 23 17:44:50.495279 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:44:50.495246    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f1014ac1-988e-47e8-b409-ececf480e8c0" containerName="revoke-keys"
Jul 23 17:44:50.495279 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:44:50.495275    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="f1014ac1-988e-47e8-b409-ececf480e8c0" containerName="revoke-keys"
Jul 23 17:44:50.495368 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:44:50.495296    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="245eb040-a556-416b-8f43-98e2ee471c51" containerName="maas-api"
Jul 23 17:44:50.495368 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:44:50.495302    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="245eb040-a556-416b-8f43-98e2ee471c51" containerName="maas-api"
Jul 23 17:44:50.495368 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:44:50.495367    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="f1014ac1-988e-47e8-b409-ececf480e8c0" containerName="revoke-keys"
Jul 23 17:44:50.495466 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:44:50.495378    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="245eb040-a556-416b-8f43-98e2ee471c51" containerName="maas-api"
Jul 23 17:44:50.498451 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:44:50.498432    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-b9d590-ffdc5bcb6-qggkj"
Jul 23 17:44:50.505895 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:44:50.505877    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-b9d590-serving-cert\""
Jul 23 17:44:50.516338 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:44:50.516312    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:44:50.535613 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:44:50.535592    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-b9d590-ffdc5bcb6-qggkj"]
Jul 23 17:44:50.593419 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:44:50.593339    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5z68\" (UniqueName: \"kubernetes.io/projected/f00d0bc9-60c8-4b62-9622-15c089d173c9-kube-api-access-f5z68\") pod \"maas-api-e2e-shared-a-b9d590-ffdc5bcb6-qggkj\" (UID: \"f00d0bc9-60c8-4b62-9622-15c089d173c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-b9d590-ffdc5bcb6-qggkj"
Jul 23 17:44:50.593419 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:44:50.593397    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f00d0bc9-60c8-4b62-9622-15c089d173c9-maas-api-tls\") pod \"maas-api-e2e-shared-a-b9d590-ffdc5bcb6-qggkj\" (UID: \"f00d0bc9-60c8-4b62-9622-15c089d173c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-b9d590-ffdc5bcb6-qggkj"
Jul 23 17:44:50.593598 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:44:50.593503    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f00d0bc9-60c8-4b62-9622-15c089d173c9-openshift-service-ca\") pod \"maas-api-e2e-shared-a-b9d590-ffdc5bcb6-qggkj\" (UID: \"f00d0bc9-60c8-4b62-9622-15c089d173c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-b9d590-ffdc5bcb6-qggkj"
Jul 23 17:44:50.622495 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:44:50.622464    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:44:50.694274 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:44:50.694224    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f00d0bc9-60c8-4b62-9622-15c089d173c9-maas-api-tls\") pod \"maas-api-e2e-shared-a-b9d590-ffdc5bcb6-qggkj\" (UID: \"f00d0bc9-60c8-4b62-9622-15c089d173c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-b9d590-ffdc5bcb6-qggkj"
Jul 23 17:44:50.694455 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:44:50.694304    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f00d0bc9-60c8-4b62-9622-15c089d173c9-openshift-service-ca\") pod \"maas-api-e2e-shared-a-b9d590-ffdc5bcb6-qggkj\" (UID: \"f00d0bc9-60c8-4b62-9622-15c089d173c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-b9d590-ffdc5bcb6-qggkj"
Jul 23 17:44:50.694455 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:44:50.694347    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5z68\" (UniqueName: \"kubernetes.io/projected/f00d0bc9-60c8-4b62-9622-15c089d173c9-kube-api-access-f5z68\") pod \"maas-api-e2e-shared-a-b9d590-ffdc5bcb6-qggkj\" (UID: \"f00d0bc9-60c8-4b62-9622-15c089d173c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-b9d590-ffdc5bcb6-qggkj"
Jul 23 17:44:50.694989 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:44:50.694962    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f00d0bc9-60c8-4b62-9622-15c089d173c9-openshift-service-ca\") pod \"maas-api-e2e-shared-a-b9d590-ffdc5bcb6-qggkj\" (UID: \"f00d0bc9-60c8-4b62-9622-15c089d173c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-b9d590-ffdc5bcb6-qggkj"
Jul 23 17:44:50.696581 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:44:50.696561    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f00d0bc9-60c8-4b62-9622-15c089d173c9-maas-api-tls\") pod \"maas-api-e2e-shared-a-b9d590-ffdc5bcb6-qggkj\" (UID: \"f00d0bc9-60c8-4b62-9622-15c089d173c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-b9d590-ffdc5bcb6-qggkj"
Jul 23 17:44:50.707807 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:44:50.707789    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5z68\" (UniqueName: \"kubernetes.io/projected/f00d0bc9-60c8-4b62-9622-15c089d173c9-kube-api-access-f5z68\") pod \"maas-api-e2e-shared-a-b9d590-ffdc5bcb6-qggkj\" (UID: \"f00d0bc9-60c8-4b62-9622-15c089d173c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-b9d590-ffdc5bcb6-qggkj"
Jul 23 17:44:50.732994 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:44:50.732966    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:44:50.808686 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:44:50.808657    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-b9d590-ffdc5bcb6-qggkj"
Jul 23 17:44:50.968392 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:44:50.968361    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf00d0bc9_60c8_4b62_9622_15c089d173c9.slice/crio-9f7be1fc080c7c89faf3ed7d447eef9aa1be307cbe7d06ea0ec3c1b0c2fbad95 WatchSource:0}: Error finding container 9f7be1fc080c7c89faf3ed7d447eef9aa1be307cbe7d06ea0ec3c1b0c2fbad95: Status 404 returned error can't find the container with id 9f7be1fc080c7c89faf3ed7d447eef9aa1be307cbe7d06ea0ec3c1b0c2fbad95
Jul 23 17:44:50.969012 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:44:50.968991    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-b9d590-ffdc5bcb6-qggkj"]
Jul 23 17:44:51.411149 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:44:51.411112    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:44:51.586001 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:44:51.585963    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:44:51.696016 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:44:51.695979    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-b9d590-ffdc5bcb6-qggkj" event={"ID":"f00d0bc9-60c8-4b62-9622-15c089d173c9","Type":"ContainerStarted","Data":"9f7be1fc080c7c89faf3ed7d447eef9aa1be307cbe7d06ea0ec3c1b0c2fbad95"}
Jul 23 17:44:52.703811 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:44:52.703778    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-b9d590-ffdc5bcb6-qggkj" event={"ID":"f00d0bc9-60c8-4b62-9622-15c089d173c9","Type":"ContainerStarted","Data":"7f6c84397209d9ce82322749fed3e58c32e38a46c71adb9365c5511c382bd30e"}
Jul 23 17:44:52.704200 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:44:52.703878    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-b9d590-ffdc5bcb6-qggkj"
Jul 23 17:44:52.723753 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:44:52.723707    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-b9d590-ffdc5bcb6-qggkj" podStartSLOduration=1.874196551 podStartE2EDuration="2.723693337s" totalImagesPullingTime="849.496786ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 17:44:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 17:44:52.721042951 +0000 UTC m=+2388.491328473" watchObservedRunningTime="2026-07-23 17:44:52.723693337 +0000 UTC m=+2388.493978890"
Jul 23 17:44:54.827069 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:44:54.827035    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:44:55.171909 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:44:55.171830    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:44:55.518195 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:44:55.518155    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:44:55.881897 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:44:55.881821    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:44:58.357636 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:44:58.357600    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-9c4020-5869f85b5b-v52vg"]
Jul 23 17:44:58.361677 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:44:58.361650    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-9c4020-5869f85b5b-v52vg"
Jul 23 17:44:58.364120 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:44:58.364093    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-9c4020-serving-cert\""
Jul 23 17:44:58.376224 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:44:58.376198    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-9c4020-5869f85b5b-v52vg"]
Jul 23 17:44:58.467241 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:44:58.467205    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rfwj8\" (UniqueName: \"kubernetes.io/projected/b73f6c46-6811-48e3-9918-4b7c2b98595d-kube-api-access-rfwj8\") pod \"maas-api-e2e-shared-b-9c4020-5869f85b5b-v52vg\" (UID: \"b73f6c46-6811-48e3-9918-4b7c2b98595d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-9c4020-5869f85b5b-v52vg"
Jul 23 17:44:58.467436 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:44:58.467272    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b73f6c46-6811-48e3-9918-4b7c2b98595d-maas-api-tls\") pod \"maas-api-e2e-shared-b-9c4020-5869f85b5b-v52vg\" (UID: \"b73f6c46-6811-48e3-9918-4b7c2b98595d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-9c4020-5869f85b5b-v52vg"
Jul 23 17:44:58.467436 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:44:58.467350    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b73f6c46-6811-48e3-9918-4b7c2b98595d-openshift-service-ca\") pod \"maas-api-e2e-shared-b-9c4020-5869f85b5b-v52vg\" (UID: \"b73f6c46-6811-48e3-9918-4b7c2b98595d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-9c4020-5869f85b5b-v52vg"
Jul 23 17:44:58.568748 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:44:58.568718    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b73f6c46-6811-48e3-9918-4b7c2b98595d-maas-api-tls\") pod \"maas-api-e2e-shared-b-9c4020-5869f85b5b-v52vg\" (UID: \"b73f6c46-6811-48e3-9918-4b7c2b98595d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-9c4020-5869f85b5b-v52vg"
Jul 23 17:44:58.568935 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:44:58.568809    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b73f6c46-6811-48e3-9918-4b7c2b98595d-openshift-service-ca\") pod \"maas-api-e2e-shared-b-9c4020-5869f85b5b-v52vg\" (UID: \"b73f6c46-6811-48e3-9918-4b7c2b98595d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-9c4020-5869f85b5b-v52vg"
Jul 23 17:44:58.568935 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:44:58.568863    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rfwj8\" (UniqueName: \"kubernetes.io/projected/b73f6c46-6811-48e3-9918-4b7c2b98595d-kube-api-access-rfwj8\") pod \"maas-api-e2e-shared-b-9c4020-5869f85b5b-v52vg\" (UID: \"b73f6c46-6811-48e3-9918-4b7c2b98595d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-9c4020-5869f85b5b-v52vg"
Jul 23 17:44:58.569546 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:44:58.569518    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b73f6c46-6811-48e3-9918-4b7c2b98595d-openshift-service-ca\") pod \"maas-api-e2e-shared-b-9c4020-5869f85b5b-v52vg\" (UID: \"b73f6c46-6811-48e3-9918-4b7c2b98595d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-9c4020-5869f85b5b-v52vg"
Jul 23 17:44:58.571352 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:44:58.571324    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b73f6c46-6811-48e3-9918-4b7c2b98595d-maas-api-tls\") pod \"maas-api-e2e-shared-b-9c4020-5869f85b5b-v52vg\" (UID: \"b73f6c46-6811-48e3-9918-4b7c2b98595d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-9c4020-5869f85b5b-v52vg"
Jul 23 17:44:58.577495 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:44:58.577469    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rfwj8\" (UniqueName: \"kubernetes.io/projected/b73f6c46-6811-48e3-9918-4b7c2b98595d-kube-api-access-rfwj8\") pod \"maas-api-e2e-shared-b-9c4020-5869f85b5b-v52vg\" (UID: \"b73f6c46-6811-48e3-9918-4b7c2b98595d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-9c4020-5869f85b5b-v52vg"
Jul 23 17:44:58.676557 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:44:58.676469    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-9c4020-5869f85b5b-v52vg"
Jul 23 17:44:58.714576 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:44:58.714538    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:44:58.715873 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:44:58.715851    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-b9d590-ffdc5bcb6-qggkj"
Jul 23 17:44:58.816508 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:44:58.816455    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:44:58.825353 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:44:58.825332    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-9c4020-5869f85b5b-v52vg"]
Jul 23 17:44:58.826821 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:44:58.826797    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb73f6c46_6811_48e3_9918_4b7c2b98595d.slice/crio-f9dc40c1122cd2bb76ef9b5a1d427ea21bc8faf99243eba989608bfc7f158251 WatchSource:0}: Error finding container f9dc40c1122cd2bb76ef9b5a1d427ea21bc8faf99243eba989608bfc7f158251: Status 404 returned error can't find the container with id f9dc40c1122cd2bb76ef9b5a1d427ea21bc8faf99243eba989608bfc7f158251
Jul 23 17:44:59.073632 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:44:59.073586    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:44:59.279776 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:44:59.279744    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:44:59.757139 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:44:59.757103    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-9c4020-5869f85b5b-v52vg" event={"ID":"b73f6c46-6811-48e3-9918-4b7c2b98595d","Type":"ContainerStarted","Data":"f9dc40c1122cd2bb76ef9b5a1d427ea21bc8faf99243eba989608bfc7f158251"}
Jul 23 17:45:00.145163 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:45:00.145077    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29747145-64pnm"]
Jul 23 17:45:00.149583 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:45:00.149559    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-b9d590-29747145-l4rqc"]
Jul 23 17:45:00.149713 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:45:00.149685    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29747145-64pnm"
Jul 23 17:45:00.152634 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:45:00.152606    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7wfmw\""
Jul 23 17:45:00.152782 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:45:00.152759    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-9c4020-29747145-bdmf6"]
Jul 23 17:45:00.152980 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:45:00.152956    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-b9d590-29747145-l4rqc"
Jul 23 17:45:00.156395 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:45:00.156376    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-9c4020-29747145-bdmf6"
Jul 23 17:45:00.159494 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:45:00.159443    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29747145-64pnm"]
Jul 23 17:45:00.164199 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:45:00.164176    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29747145-64pnm"
Jul 23 17:45:00.171473 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:45:00.171441    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-9c4020-29747145-bdmf6"]
Jul 23 17:45:00.175493 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:45:00.175472    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-b9d590-29747145-l4rqc"]
Jul 23 17:45:00.176777 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:45:00.176621    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-b9d590-29747145-l4rqc"
Jul 23 17:45:00.212107 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:45:00.212084    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-9c4020-29747145-bdmf6"
Jul 23 17:45:00.338003 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:45:00.337954    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29747145-64pnm"]
Jul 23 17:45:00.344930 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:45:00.344885    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78cb17f5_738b_40f1_b40b_458ed15c8b06.slice/crio-d32cf079af9cfab6a8de647a43ae7ce9a13a1bc2d749ed8c0c5074e0654f581b WatchSource:0}: Error finding container d32cf079af9cfab6a8de647a43ae7ce9a13a1bc2d749ed8c0c5074e0654f581b: Status 404 returned error can't find the container with id d32cf079af9cfab6a8de647a43ae7ce9a13a1bc2d749ed8c0c5074e0654f581b
Jul 23 17:45:00.373494 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:45:00.373468    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-b9d590-29747145-l4rqc"]
Jul 23 17:45:00.380404 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:45:00.380376    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41038e83_4fdc_4c45_a443_b0778b85e7b0.slice/crio-851c17c28a2cd98db94991ccf3b9e8c1cc36b5d240dcad59454a822c457c5a4e WatchSource:0}: Error finding container 851c17c28a2cd98db94991ccf3b9e8c1cc36b5d240dcad59454a822c457c5a4e: Status 404 returned error can't find the container with id 851c17c28a2cd98db94991ccf3b9e8c1cc36b5d240dcad59454a822c457c5a4e
Jul 23 17:45:00.399761 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:45:00.399732    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-9c4020-29747145-bdmf6"]
Jul 23 17:45:00.400860 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:45:00.400838    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41eff2f9_9ec2_4038_bcdc_8674fd6e2a3c.slice/crio-3848c641bca40c63ebed88cb99c1a629e743d5305c0f89bdf65d1d94b56829c3 WatchSource:0}: Error finding container 3848c641bca40c63ebed88cb99c1a629e743d5305c0f89bdf65d1d94b56829c3: Status 404 returned error can't find the container with id 3848c641bca40c63ebed88cb99c1a629e743d5305c0f89bdf65d1d94b56829c3
Jul 23 17:45:00.766353 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:45:00.766309    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29747145-64pnm" event={"ID":"78cb17f5-738b-40f1-b40b-458ed15c8b06","Type":"ContainerStarted","Data":"cd8a618e69ae7fe6a119379b30ccecf8537ab123ce276dc3425ca5e1bad3d53b"}
Jul 23 17:45:00.766712 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:45:00.766384    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29747145-64pnm" event={"ID":"78cb17f5-738b-40f1-b40b-458ed15c8b06","Type":"ContainerStarted","Data":"d32cf079af9cfab6a8de647a43ae7ce9a13a1bc2d749ed8c0c5074e0654f581b"}
Jul 23 17:45:00.766712 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:45:00.766416    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-9c4020-5869f85b5b-v52vg"
Jul 23 17:45:00.766712 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:45:00.766446    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-9c4020-5869f85b5b-v52vg" event={"ID":"b73f6c46-6811-48e3-9918-4b7c2b98595d","Type":"ContainerStarted","Data":"dbfd8e1e67487bdff231ce897e5c54c292f9c865385a096e93a9056d287c9936"}
Jul 23 17:45:00.767448 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:45:00.767425    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-b9d590-29747145-l4rqc" event={"ID":"41038e83-4fdc-4c45-a443-b0778b85e7b0","Type":"ContainerStarted","Data":"c06d592d5414d7d37b9ed5e83972f4334db60dca8a865f57e8b16469754156fe"}
Jul 23 17:45:00.767556 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:45:00.767462    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-b9d590-29747145-l4rqc" event={"ID":"41038e83-4fdc-4c45-a443-b0778b85e7b0","Type":"ContainerStarted","Data":"851c17c28a2cd98db94991ccf3b9e8c1cc36b5d240dcad59454a822c457c5a4e"}
Jul 23 17:45:00.768802 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:45:00.768782    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-9c4020-29747145-bdmf6" event={"ID":"41eff2f9-9ec2-4038-bcdc-8674fd6e2a3c","Type":"ContainerStarted","Data":"633af1c861ec20be88f7e1190631e6b1ade39d6c9e532468c15b26536277079e"}
Jul 23 17:45:00.768919 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:45:00.768816    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-9c4020-29747145-bdmf6" event={"ID":"41eff2f9-9ec2-4038-bcdc-8674fd6e2a3c","Type":"ContainerStarted","Data":"3848c641bca40c63ebed88cb99c1a629e743d5305c0f89bdf65d1d94b56829c3"}
Jul 23 17:45:00.781834 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:45:00.781786    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29747145-64pnm" podStartSLOduration=0.781772917 podStartE2EDuration="781.772917ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 17:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 17:45:00.779732435 +0000 UTC m=+2396.550017946" watchObservedRunningTime="2026-07-23 17:45:00.781772917 +0000 UTC m=+2396.552058436"
Jul 23 17:45:00.796816 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:45:00.796775    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-b9d590-29747145-l4rqc" podStartSLOduration=0.796763594 podStartE2EDuration="796.763594ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 17:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 17:45:00.795430356 +0000 UTC m=+2396.565715892" watchObservedRunningTime="2026-07-23 17:45:00.796763594 +0000 UTC m=+2396.567049113"
Jul 23 17:45:00.816982 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:45:00.816940    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-9c4020-29747145-bdmf6" podStartSLOduration=0.816929622 podStartE2EDuration="816.929622ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 17:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 17:45:00.813730007 +0000 UTC m=+2396.584015528" watchObservedRunningTime="2026-07-23 17:45:00.816929622 +0000 UTC m=+2396.587215141"
Jul 23 17:45:00.840024 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:45:00.839974    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-9c4020-5869f85b5b-v52vg" podStartSLOduration=1.877859955 podStartE2EDuration="2.839958125s" totalImagesPullingTime="962.09817ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 17:44:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 17:45:00.838316685 +0000 UTC m=+2396.608602204" watchObservedRunningTime="2026-07-23 17:45:00.839958125 +0000 UTC m=+2396.610243648"
Jul 23 17:45:01.776186 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:45:01.776151    2577 generic.go:356] "Generic (PLEG): container finished" podID="78cb17f5-738b-40f1-b40b-458ed15c8b06" containerID="cd8a618e69ae7fe6a119379b30ccecf8537ab123ce276dc3425ca5e1bad3d53b" exitCode=0
Jul 23 17:45:01.776672 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:45:01.776236    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29747145-64pnm" event={"ID":"78cb17f5-738b-40f1-b40b-458ed15c8b06","Type":"ContainerDied","Data":"cd8a618e69ae7fe6a119379b30ccecf8537ab123ce276dc3425ca5e1bad3d53b"}
Jul 23 17:45:01.777970 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:45:01.777948    2577 generic.go:356] "Generic (PLEG): container finished" podID="41038e83-4fdc-4c45-a443-b0778b85e7b0" containerID="c06d592d5414d7d37b9ed5e83972f4334db60dca8a865f57e8b16469754156fe" exitCode=0
Jul 23 17:45:01.778117 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:45:01.778019    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-b9d590-29747145-l4rqc" event={"ID":"41038e83-4fdc-4c45-a443-b0778b85e7b0","Type":"ContainerDied","Data":"c06d592d5414d7d37b9ed5e83972f4334db60dca8a865f57e8b16469754156fe"}
Jul 23 17:45:01.779502 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:45:01.779481    2577 generic.go:356] "Generic (PLEG): container finished" podID="41eff2f9-9ec2-4038-bcdc-8674fd6e2a3c" containerID="633af1c861ec20be88f7e1190631e6b1ade39d6c9e532468c15b26536277079e" exitCode=7
Jul 23 17:45:01.779596 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:45:01.779550    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-9c4020-29747145-bdmf6" event={"ID":"41eff2f9-9ec2-4038-bcdc-8674fd6e2a3c","Type":"ContainerDied","Data":"633af1c861ec20be88f7e1190631e6b1ade39d6c9e532468c15b26536277079e"}
Jul 23 17:45:01.779700 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:45:01.779682    2577 scope.go:117] "RemoveContainer" containerID="633af1c861ec20be88f7e1190631e6b1ade39d6c9e532468c15b26536277079e"
Jul 23 17:45:02.786892 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:45:02.786853    2577 generic.go:356] "Generic (PLEG): container finished" podID="41eff2f9-9ec2-4038-bcdc-8674fd6e2a3c" containerID="094e884739d4f8e4889260271f06c9973efd440eece643f889d7979c2ee790d7" exitCode=7
Jul 23 17:45:02.787403 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:45:02.786942    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-9c4020-29747145-bdmf6" event={"ID":"41eff2f9-9ec2-4038-bcdc-8674fd6e2a3c","Type":"ContainerDied","Data":"094e884739d4f8e4889260271f06c9973efd440eece643f889d7979c2ee790d7"}
Jul 23 17:45:02.787403 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:45:02.787004    2577 scope.go:117] "RemoveContainer" containerID="633af1c861ec20be88f7e1190631e6b1ade39d6c9e532468c15b26536277079e"
Jul 23 17:45:02.787403 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:45:02.787214    2577 scope.go:117] "RemoveContainer" containerID="094e884739d4f8e4889260271f06c9973efd440eece643f889d7979c2ee790d7"
Jul 23 17:45:02.787651 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:45:02.787511    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-9c4020-29747145-bdmf6_odh-ai-gateway-infra(41eff2f9-9ec2-4038-bcdc-8674fd6e2a3c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-9c4020-29747145-bdmf6" podUID="41eff2f9-9ec2-4038-bcdc-8674fd6e2a3c"
Jul 23 17:45:02.938354 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:45:02.938320    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29747145-64pnm"
Jul 23 17:45:02.970631 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:45:02.970603    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-b9d590-29747145-l4rqc"
Jul 23 17:45:03.582798 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:45:03.582764    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:45:03.800390 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:45:03.800361    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-b9d590-29747145-l4rqc"
Jul 23 17:45:03.800842 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:45:03.800358    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-b9d590-29747145-l4rqc" event={"ID":"41038e83-4fdc-4c45-a443-b0778b85e7b0","Type":"ContainerDied","Data":"851c17c28a2cd98db94991ccf3b9e8c1cc36b5d240dcad59454a822c457c5a4e"}
Jul 23 17:45:03.800842 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:45:03.800507    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="851c17c28a2cd98db94991ccf3b9e8c1cc36b5d240dcad59454a822c457c5a4e"
Jul 23 17:45:03.802227 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:45:03.802201    2577 scope.go:117] "RemoveContainer" containerID="094e884739d4f8e4889260271f06c9973efd440eece643f889d7979c2ee790d7"
Jul 23 17:45:03.802434 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:45:03.802414    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-9c4020-29747145-bdmf6_odh-ai-gateway-infra(41eff2f9-9ec2-4038-bcdc-8674fd6e2a3c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-9c4020-29747145-bdmf6" podUID="41eff2f9-9ec2-4038-bcdc-8674fd6e2a3c"
Jul 23 17:45:03.803507 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:45:03.803484    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29747145-64pnm"
Jul 23 17:45:03.803588 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:45:03.803531    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29747145-64pnm" event={"ID":"78cb17f5-738b-40f1-b40b-458ed15c8b06","Type":"ContainerDied","Data":"d32cf079af9cfab6a8de647a43ae7ce9a13a1bc2d749ed8c0c5074e0654f581b"}
Jul 23 17:45:03.803588 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:45:03.803566    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d32cf079af9cfab6a8de647a43ae7ce9a13a1bc2d749ed8c0c5074e0654f581b"
Jul 23 17:45:06.785566 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:45:06.785532    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-9c4020-5869f85b5b-v52vg"
Jul 23 17:45:11.326044 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:45:11.326004    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:45:12.756247 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:45:12.756216    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:45:12.859625 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:45:12.859582    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:45:16.760041 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:45:16.760011    2577 scope.go:117] "RemoveContainer" containerID="094e884739d4f8e4889260271f06c9973efd440eece643f889d7979c2ee790d7"
Jul 23 17:45:17.887728 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:45:17.887689    2577 generic.go:356] "Generic (PLEG): container finished" podID="41eff2f9-9ec2-4038-bcdc-8674fd6e2a3c" containerID="783cf0c85dff80a48f68d488b05a0b4d868a6663b08ebf51dd555672d7210a69" exitCode=0
Jul 23 17:45:17.888134 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:45:17.887763    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-9c4020-29747145-bdmf6" event={"ID":"41eff2f9-9ec2-4038-bcdc-8674fd6e2a3c","Type":"ContainerDied","Data":"783cf0c85dff80a48f68d488b05a0b4d868a6663b08ebf51dd555672d7210a69"}
Jul 23 17:45:17.888134 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:45:17.887825    2577 scope.go:117] "RemoveContainer" containerID="094e884739d4f8e4889260271f06c9973efd440eece643f889d7979c2ee790d7"
Jul 23 17:45:18.922817 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:45:18.922789    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-9c4020-29747145-bdmf6"]
Jul 23 17:45:19.030789 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:45:19.030759    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-9c4020-29747145-bdmf6"
Jul 23 17:45:19.903927 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:45:19.903897    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-9c4020-29747145-bdmf6" event={"ID":"41eff2f9-9ec2-4038-bcdc-8674fd6e2a3c","Type":"ContainerDied","Data":"3848c641bca40c63ebed88cb99c1a629e743d5305c0f89bdf65d1d94b56829c3"}
Jul 23 17:45:19.904098 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:45:19.903943    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3848c641bca40c63ebed88cb99c1a629e743d5305c0f89bdf65d1d94b56829c3"
Jul 23 17:45:19.904098 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:45:19.903906    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-9c4020-29747145-bdmf6"
Jul 23 17:45:19.948960 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:45:19.948930    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-9c4020-29747145-bdmf6"]
Jul 23 17:45:19.957483 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:45:19.957459    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-9c4020-29747145-bdmf6"]
Jul 23 17:45:20.766193 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:45:20.766159    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="41eff2f9-9ec2-4038-bcdc-8674fd6e2a3c" path="/var/lib/kubelet/pods/41eff2f9-9ec2-4038-bcdc-8674fd6e2a3c/volumes"
Jul 23 17:45:22.581880 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:45:22.581850    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:45:22.681553 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:45:22.681508    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:45:47.596902 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:45:47.596866    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-9c4020/auth-test-model-9c4020-kserve-6787fd5787-b2jhf"]
Jul 23 17:45:47.597353 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:45:47.597315    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41eff2f9-9ec2-4038-bcdc-8674fd6e2a3c" containerName="cleanup"
Jul 23 17:45:47.597353 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:45:47.597328    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="41eff2f9-9ec2-4038-bcdc-8674fd6e2a3c" containerName="cleanup"
Jul 23 17:45:47.597353 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:45:47.597339    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41eff2f9-9ec2-4038-bcdc-8674fd6e2a3c" containerName="cleanup"
Jul 23 17:45:47.597353 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:45:47.597344    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="41eff2f9-9ec2-4038-bcdc-8674fd6e2a3c" containerName="cleanup"
Jul 23 17:45:47.597518 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:45:47.597362    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="78cb17f5-738b-40f1-b40b-458ed15c8b06" containerName="cleanup"
Jul 23 17:45:47.597518 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:45:47.597368    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="78cb17f5-738b-40f1-b40b-458ed15c8b06" containerName="cleanup"
Jul 23 17:45:47.597518 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:45:47.597389    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41038e83-4fdc-4c45-a443-b0778b85e7b0" containerName="cleanup"
Jul 23 17:45:47.597518 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:45:47.597394    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="41038e83-4fdc-4c45-a443-b0778b85e7b0" containerName="cleanup"
Jul 23 17:45:47.597518 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:45:47.597463    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="41eff2f9-9ec2-4038-bcdc-8674fd6e2a3c" containerName="cleanup"
Jul 23 17:45:47.597518 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:45:47.597470    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="78cb17f5-738b-40f1-b40b-458ed15c8b06" containerName="cleanup"
Jul 23 17:45:47.597518 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:45:47.597479    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="41038e83-4fdc-4c45-a443-b0778b85e7b0" containerName="cleanup"
Jul 23 17:45:47.597518 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:45:47.597487    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="41eff2f9-9ec2-4038-bcdc-8674fd6e2a3c" containerName="cleanup"
Jul 23 17:45:47.601603 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:45:47.601582    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-9c4020/auth-test-model-9c4020-kserve-6787fd5787-b2jhf"
Jul 23 17:45:47.604812 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:45:47.604778    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-9c4020\"/\"auth-test-model-9c4020-kserve-self-signed-certs\""
Jul 23 17:45:47.604812 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:45:47.604801    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-9c4020\"/\"kube-root-ca.crt\""
Jul 23 17:45:47.605029 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:45:47.604782    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-9c4020\"/\"default-dockercfg-hvqd9\""
Jul 23 17:45:47.605029 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:45:47.604795    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-9c4020\"/\"openshift-service-ca.crt\""
Jul 23 17:45:47.610205 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:45:47.610179    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-9c4020/auth-test-model-9c4020-kserve-6787fd5787-b2jhf"]
Jul 23 17:45:47.632242 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:45:47.632214    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:45:47.728710 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:45:47.728681    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/90d33a4a-220d-40a3-a344-6a6b139eb67d-dshm\") pod \"auth-test-model-9c4020-kserve-6787fd5787-b2jhf\" (UID: \"90d33a4a-220d-40a3-a344-6a6b139eb67d\") " pod="ai-tenant-e2e-shared-b-9c4020/auth-test-model-9c4020-kserve-6787fd5787-b2jhf"
Jul 23 17:45:47.728856 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:45:47.728734    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/90d33a4a-220d-40a3-a344-6a6b139eb67d-home\") pod \"auth-test-model-9c4020-kserve-6787fd5787-b2jhf\" (UID: \"90d33a4a-220d-40a3-a344-6a6b139eb67d\") " pod="ai-tenant-e2e-shared-b-9c4020/auth-test-model-9c4020-kserve-6787fd5787-b2jhf"
Jul 23 17:45:47.728856 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:45:47.728788    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/90d33a4a-220d-40a3-a344-6a6b139eb67d-model-cache\") pod \"auth-test-model-9c4020-kserve-6787fd5787-b2jhf\" (UID: \"90d33a4a-220d-40a3-a344-6a6b139eb67d\") " pod="ai-tenant-e2e-shared-b-9c4020/auth-test-model-9c4020-kserve-6787fd5787-b2jhf"
Jul 23 17:45:47.728856 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:45:47.728853    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90d33a4a-220d-40a3-a344-6a6b139eb67d-tls-certs\") pod \"auth-test-model-9c4020-kserve-6787fd5787-b2jhf\" (UID: \"90d33a4a-220d-40a3-a344-6a6b139eb67d\") " pod="ai-tenant-e2e-shared-b-9c4020/auth-test-model-9c4020-kserve-6787fd5787-b2jhf"
Jul 23 17:45:47.728965 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:45:47.728872    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90d33a4a-220d-40a3-a344-6a6b139eb67d-tmp-dir\") pod \"auth-test-model-9c4020-kserve-6787fd5787-b2jhf\" (UID: \"90d33a4a-220d-40a3-a344-6a6b139eb67d\") " pod="ai-tenant-e2e-shared-b-9c4020/auth-test-model-9c4020-kserve-6787fd5787-b2jhf"
Jul 23 17:45:47.728965 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:45:47.728897    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2wlkk\" (UniqueName: \"kubernetes.io/projected/90d33a4a-220d-40a3-a344-6a6b139eb67d-kube-api-access-2wlkk\") pod \"auth-test-model-9c4020-kserve-6787fd5787-b2jhf\" (UID: \"90d33a4a-220d-40a3-a344-6a6b139eb67d\") " pod="ai-tenant-e2e-shared-b-9c4020/auth-test-model-9c4020-kserve-6787fd5787-b2jhf"
Jul 23 17:45:47.829419 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:45:47.829386    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/90d33a4a-220d-40a3-a344-6a6b139eb67d-dshm\") pod \"auth-test-model-9c4020-kserve-6787fd5787-b2jhf\" (UID: \"90d33a4a-220d-40a3-a344-6a6b139eb67d\") " pod="ai-tenant-e2e-shared-b-9c4020/auth-test-model-9c4020-kserve-6787fd5787-b2jhf"
Jul 23 17:45:47.829605 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:45:47.829455    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/90d33a4a-220d-40a3-a344-6a6b139eb67d-home\") pod \"auth-test-model-9c4020-kserve-6787fd5787-b2jhf\" (UID: \"90d33a4a-220d-40a3-a344-6a6b139eb67d\") " pod="ai-tenant-e2e-shared-b-9c4020/auth-test-model-9c4020-kserve-6787fd5787-b2jhf"
Jul 23 17:45:47.829605 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:45:47.829479    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/90d33a4a-220d-40a3-a344-6a6b139eb67d-model-cache\") pod \"auth-test-model-9c4020-kserve-6787fd5787-b2jhf\" (UID: \"90d33a4a-220d-40a3-a344-6a6b139eb67d\") " pod="ai-tenant-e2e-shared-b-9c4020/auth-test-model-9c4020-kserve-6787fd5787-b2jhf"
Jul 23 17:45:47.829605 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:45:47.829509    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90d33a4a-220d-40a3-a344-6a6b139eb67d-tls-certs\") pod \"auth-test-model-9c4020-kserve-6787fd5787-b2jhf\" (UID: \"90d33a4a-220d-40a3-a344-6a6b139eb67d\") " pod="ai-tenant-e2e-shared-b-9c4020/auth-test-model-9c4020-kserve-6787fd5787-b2jhf"
Jul 23 17:45:47.829605 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:45:47.829538    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90d33a4a-220d-40a3-a344-6a6b139eb67d-tmp-dir\") pod \"auth-test-model-9c4020-kserve-6787fd5787-b2jhf\" (UID: \"90d33a4a-220d-40a3-a344-6a6b139eb67d\") " pod="ai-tenant-e2e-shared-b-9c4020/auth-test-model-9c4020-kserve-6787fd5787-b2jhf"
Jul 23 17:45:47.829605 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:45:47.829574    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2wlkk\" (UniqueName: \"kubernetes.io/projected/90d33a4a-220d-40a3-a344-6a6b139eb67d-kube-api-access-2wlkk\") pod \"auth-test-model-9c4020-kserve-6787fd5787-b2jhf\" (UID: \"90d33a4a-220d-40a3-a344-6a6b139eb67d\") " pod="ai-tenant-e2e-shared-b-9c4020/auth-test-model-9c4020-kserve-6787fd5787-b2jhf"
Jul 23 17:45:47.829872 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:45:47.829854    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/90d33a4a-220d-40a3-a344-6a6b139eb67d-home\") pod \"auth-test-model-9c4020-kserve-6787fd5787-b2jhf\" (UID: \"90d33a4a-220d-40a3-a344-6a6b139eb67d\") " pod="ai-tenant-e2e-shared-b-9c4020/auth-test-model-9c4020-kserve-6787fd5787-b2jhf"
Jul 23 17:45:47.829926 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:45:47.829894    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/90d33a4a-220d-40a3-a344-6a6b139eb67d-model-cache\") pod \"auth-test-model-9c4020-kserve-6787fd5787-b2jhf\" (UID: \"90d33a4a-220d-40a3-a344-6a6b139eb67d\") " pod="ai-tenant-e2e-shared-b-9c4020/auth-test-model-9c4020-kserve-6787fd5787-b2jhf"
Jul 23 17:45:47.829996 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:45:47.829933    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90d33a4a-220d-40a3-a344-6a6b139eb67d-tmp-dir\") pod \"auth-test-model-9c4020-kserve-6787fd5787-b2jhf\" (UID: \"90d33a4a-220d-40a3-a344-6a6b139eb67d\") " pod="ai-tenant-e2e-shared-b-9c4020/auth-test-model-9c4020-kserve-6787fd5787-b2jhf"
Jul 23 17:45:47.831954 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:45:47.831932    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/90d33a4a-220d-40a3-a344-6a6b139eb67d-dshm\") pod \"auth-test-model-9c4020-kserve-6787fd5787-b2jhf\" (UID: \"90d33a4a-220d-40a3-a344-6a6b139eb67d\") " pod="ai-tenant-e2e-shared-b-9c4020/auth-test-model-9c4020-kserve-6787fd5787-b2jhf"
Jul 23 17:45:47.831954 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:45:47.831942    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90d33a4a-220d-40a3-a344-6a6b139eb67d-tls-certs\") pod \"auth-test-model-9c4020-kserve-6787fd5787-b2jhf\" (UID: \"90d33a4a-220d-40a3-a344-6a6b139eb67d\") " pod="ai-tenant-e2e-shared-b-9c4020/auth-test-model-9c4020-kserve-6787fd5787-b2jhf"
Jul 23 17:45:47.843836 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:45:47.843816    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2wlkk\" (UniqueName: \"kubernetes.io/projected/90d33a4a-220d-40a3-a344-6a6b139eb67d-kube-api-access-2wlkk\") pod \"auth-test-model-9c4020-kserve-6787fd5787-b2jhf\" (UID: \"90d33a4a-220d-40a3-a344-6a6b139eb67d\") " pod="ai-tenant-e2e-shared-b-9c4020/auth-test-model-9c4020-kserve-6787fd5787-b2jhf"
Jul 23 17:45:47.912622 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:45:47.912550    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-9c4020/auth-test-model-9c4020-kserve-6787fd5787-b2jhf"
Jul 23 17:45:48.056568 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:45:48.056530    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-9c4020/auth-test-model-9c4020-kserve-6787fd5787-b2jhf"]
Jul 23 17:45:48.057866 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:45:48.057842    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod90d33a4a_220d_40a3_a344_6a6b139eb67d.slice/crio-66dbddf9529473776bad932286372f2e9d7007bbedff39d58489633a5ce18769 WatchSource:0}: Error finding container 66dbddf9529473776bad932286372f2e9d7007bbedff39d58489633a5ce18769: Status 404 returned error can't find the container with id 66dbddf9529473776bad932286372f2e9d7007bbedff39d58489633a5ce18769
Jul 23 17:45:49.071787 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:45:49.071754    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-9c4020/auth-test-model-9c4020-kserve-6787fd5787-b2jhf" event={"ID":"90d33a4a-220d-40a3-a344-6a6b139eb67d","Type":"ContainerStarted","Data":"4f6dda3bf50f7aa9d83fe19d28b966acc89e7a13c313b4b1899cd0b8b5f4f92a"}
Jul 23 17:45:49.072176 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:45:49.071798    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-9c4020/auth-test-model-9c4020-kserve-6787fd5787-b2jhf" event={"ID":"90d33a4a-220d-40a3-a344-6a6b139eb67d","Type":"ContainerStarted","Data":"66dbddf9529473776bad932286372f2e9d7007bbedff39d58489633a5ce18769"}
Jul 23 17:45:49.092645 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:45:49.092597    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-9c4020/auth-test-model-9c4020-kserve-6787fd5787-b2jhf" podStartSLOduration=2.092582281 podStartE2EDuration="2.092582281s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 17:45:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 17:45:49.08938642 +0000 UTC m=+2444.859671953" watchObservedRunningTime="2026-07-23 17:45:49.092582281 +0000 UTC m=+2444.862867800"
Jul 23 17:45:57.912695 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:45:57.912659    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-9c4020/auth-test-model-9c4020-kserve-6787fd5787-b2jhf"
Jul 23 17:45:57.913319 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:45:57.912717    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-9c4020/auth-test-model-9c4020-kserve-6787fd5787-b2jhf"
Jul 23 17:45:57.925338 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:45:57.925314    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-9c4020/auth-test-model-9c4020-kserve-6787fd5787-b2jhf"
Jul 23 17:45:58.134608 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:45:58.134574    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-9c4020/auth-test-model-9c4020-kserve-6787fd5787-b2jhf"
Jul 23 17:47:23.981448 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:47:23.981368    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:47:25.637105 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:47:25.637073    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:47:31.434026 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:47:31.433985    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:47:31.534940 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:47:31.534900    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:48:01.537746 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:48:01.537701    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:48:02.303351 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:48:02.303310    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-9c4020/auth-test-model-9c4020-kserve-6787fd5787-b2jhf"]
Jul 23 17:48:02.321933 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:48:02.321892    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:48:12.801987 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:48:12.801956    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-b9d590/sub-test-model-b9d590-kserve-9447f7bfc-p48xw"]
Jul 23 17:48:12.802432 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:48:12.802418    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41eff2f9-9ec2-4038-bcdc-8674fd6e2a3c" containerName="cleanup"
Jul 23 17:48:12.802432 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:48:12.802433    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="41eff2f9-9ec2-4038-bcdc-8674fd6e2a3c" containerName="cleanup"
Jul 23 17:48:12.802517 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:48:12.802500    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="41eff2f9-9ec2-4038-bcdc-8674fd6e2a3c" containerName="cleanup"
Jul 23 17:48:12.807477 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:48:12.807460    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-b9d590/sub-test-model-b9d590-kserve-9447f7bfc-p48xw"
Jul 23 17:48:12.812027 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:48:12.811941    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-b9d590\"/\"sub-test-model-b9d590-kserve-self-signed-certs\""
Jul 23 17:48:12.812027 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:48:12.811956    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-b9d590\"/\"openshift-service-ca.crt\""
Jul 23 17:48:12.812027 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:48:12.811951    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-b9d590\"/\"kube-root-ca.crt\""
Jul 23 17:48:12.812484 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:48:12.812199    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-b9d590\"/\"default-dockercfg-d2fpk\""
Jul 23 17:48:12.817559 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:48:12.817538    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-b9d590/sub-test-model-b9d590-kserve-9447f7bfc-p48xw"]
Jul 23 17:48:12.859179 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:48:12.859142    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/821216c1-086f-4287-80e2-be36b4bdc07f-model-cache\") pod \"sub-test-model-b9d590-kserve-9447f7bfc-p48xw\" (UID: \"821216c1-086f-4287-80e2-be36b4bdc07f\") " pod="ai-tenant-e2e-shared-a-b9d590/sub-test-model-b9d590-kserve-9447f7bfc-p48xw"
Jul 23 17:48:12.859179 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:48:12.859180    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vjsd\" (UniqueName: \"kubernetes.io/projected/821216c1-086f-4287-80e2-be36b4bdc07f-kube-api-access-2vjsd\") pod \"sub-test-model-b9d590-kserve-9447f7bfc-p48xw\" (UID: \"821216c1-086f-4287-80e2-be36b4bdc07f\") " pod="ai-tenant-e2e-shared-a-b9d590/sub-test-model-b9d590-kserve-9447f7bfc-p48xw"
Jul 23 17:48:12.859419 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:48:12.859198    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/821216c1-086f-4287-80e2-be36b4bdc07f-dshm\") pod \"sub-test-model-b9d590-kserve-9447f7bfc-p48xw\" (UID: \"821216c1-086f-4287-80e2-be36b4bdc07f\") " pod="ai-tenant-e2e-shared-a-b9d590/sub-test-model-b9d590-kserve-9447f7bfc-p48xw"
Jul 23 17:48:12.859419 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:48:12.859309    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/821216c1-086f-4287-80e2-be36b4bdc07f-tmp-dir\") pod \"sub-test-model-b9d590-kserve-9447f7bfc-p48xw\" (UID: \"821216c1-086f-4287-80e2-be36b4bdc07f\") " pod="ai-tenant-e2e-shared-a-b9d590/sub-test-model-b9d590-kserve-9447f7bfc-p48xw"
Jul 23 17:48:12.859419 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:48:12.859366    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/821216c1-086f-4287-80e2-be36b4bdc07f-home\") pod \"sub-test-model-b9d590-kserve-9447f7bfc-p48xw\" (UID: \"821216c1-086f-4287-80e2-be36b4bdc07f\") " pod="ai-tenant-e2e-shared-a-b9d590/sub-test-model-b9d590-kserve-9447f7bfc-p48xw"
Jul 23 17:48:12.859419 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:48:12.859398    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/821216c1-086f-4287-80e2-be36b4bdc07f-tls-certs\") pod \"sub-test-model-b9d590-kserve-9447f7bfc-p48xw\" (UID: \"821216c1-086f-4287-80e2-be36b4bdc07f\") " pod="ai-tenant-e2e-shared-a-b9d590/sub-test-model-b9d590-kserve-9447f7bfc-p48xw"
Jul 23 17:48:12.960506 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:48:12.960466    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/821216c1-086f-4287-80e2-be36b4bdc07f-tls-certs\") pod \"sub-test-model-b9d590-kserve-9447f7bfc-p48xw\" (UID: \"821216c1-086f-4287-80e2-be36b4bdc07f\") " pod="ai-tenant-e2e-shared-a-b9d590/sub-test-model-b9d590-kserve-9447f7bfc-p48xw"
Jul 23 17:48:12.960687 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:48:12.960546    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/821216c1-086f-4287-80e2-be36b4bdc07f-model-cache\") pod \"sub-test-model-b9d590-kserve-9447f7bfc-p48xw\" (UID: \"821216c1-086f-4287-80e2-be36b4bdc07f\") " pod="ai-tenant-e2e-shared-a-b9d590/sub-test-model-b9d590-kserve-9447f7bfc-p48xw"
Jul 23 17:48:12.960687 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:48:12.960571    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vjsd\" (UniqueName: \"kubernetes.io/projected/821216c1-086f-4287-80e2-be36b4bdc07f-kube-api-access-2vjsd\") pod \"sub-test-model-b9d590-kserve-9447f7bfc-p48xw\" (UID: \"821216c1-086f-4287-80e2-be36b4bdc07f\") " pod="ai-tenant-e2e-shared-a-b9d590/sub-test-model-b9d590-kserve-9447f7bfc-p48xw"
Jul 23 17:48:12.960687 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:48:12.960590    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/821216c1-086f-4287-80e2-be36b4bdc07f-dshm\") pod \"sub-test-model-b9d590-kserve-9447f7bfc-p48xw\" (UID: \"821216c1-086f-4287-80e2-be36b4bdc07f\") " pod="ai-tenant-e2e-shared-a-b9d590/sub-test-model-b9d590-kserve-9447f7bfc-p48xw"
Jul 23 17:48:12.960687 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:48:12.960612    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/821216c1-086f-4287-80e2-be36b4bdc07f-tmp-dir\") pod \"sub-test-model-b9d590-kserve-9447f7bfc-p48xw\" (UID: \"821216c1-086f-4287-80e2-be36b4bdc07f\") " pod="ai-tenant-e2e-shared-a-b9d590/sub-test-model-b9d590-kserve-9447f7bfc-p48xw"
Jul 23 17:48:12.960687 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:48:12.960649    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/821216c1-086f-4287-80e2-be36b4bdc07f-home\") pod \"sub-test-model-b9d590-kserve-9447f7bfc-p48xw\" (UID: \"821216c1-086f-4287-80e2-be36b4bdc07f\") " pod="ai-tenant-e2e-shared-a-b9d590/sub-test-model-b9d590-kserve-9447f7bfc-p48xw"
Jul 23 17:48:12.961029 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:48:12.961005    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/821216c1-086f-4287-80e2-be36b4bdc07f-model-cache\") pod \"sub-test-model-b9d590-kserve-9447f7bfc-p48xw\" (UID: \"821216c1-086f-4287-80e2-be36b4bdc07f\") " pod="ai-tenant-e2e-shared-a-b9d590/sub-test-model-b9d590-kserve-9447f7bfc-p48xw"
Jul 23 17:48:12.961111 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:48:12.961073    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/821216c1-086f-4287-80e2-be36b4bdc07f-tmp-dir\") pod \"sub-test-model-b9d590-kserve-9447f7bfc-p48xw\" (UID: \"821216c1-086f-4287-80e2-be36b4bdc07f\") " pod="ai-tenant-e2e-shared-a-b9d590/sub-test-model-b9d590-kserve-9447f7bfc-p48xw"
Jul 23 17:48:12.961297 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:48:12.961086    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/821216c1-086f-4287-80e2-be36b4bdc07f-home\") pod \"sub-test-model-b9d590-kserve-9447f7bfc-p48xw\" (UID: \"821216c1-086f-4287-80e2-be36b4bdc07f\") " pod="ai-tenant-e2e-shared-a-b9d590/sub-test-model-b9d590-kserve-9447f7bfc-p48xw"
Jul 23 17:48:12.962840 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:48:12.962816    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/821216c1-086f-4287-80e2-be36b4bdc07f-dshm\") pod \"sub-test-model-b9d590-kserve-9447f7bfc-p48xw\" (UID: \"821216c1-086f-4287-80e2-be36b4bdc07f\") " pod="ai-tenant-e2e-shared-a-b9d590/sub-test-model-b9d590-kserve-9447f7bfc-p48xw"
Jul 23 17:48:12.963203 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:48:12.963186    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/821216c1-086f-4287-80e2-be36b4bdc07f-tls-certs\") pod \"sub-test-model-b9d590-kserve-9447f7bfc-p48xw\" (UID: \"821216c1-086f-4287-80e2-be36b4bdc07f\") " pod="ai-tenant-e2e-shared-a-b9d590/sub-test-model-b9d590-kserve-9447f7bfc-p48xw"
Jul 23 17:48:12.969700 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:48:12.969680    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vjsd\" (UniqueName: \"kubernetes.io/projected/821216c1-086f-4287-80e2-be36b4bdc07f-kube-api-access-2vjsd\") pod \"sub-test-model-b9d590-kserve-9447f7bfc-p48xw\" (UID: \"821216c1-086f-4287-80e2-be36b4bdc07f\") " pod="ai-tenant-e2e-shared-a-b9d590/sub-test-model-b9d590-kserve-9447f7bfc-p48xw"
Jul 23 17:48:13.050486 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:48:13.050461    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:48:13.118487 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:48:13.118421    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-b9d590/sub-test-model-b9d590-kserve-9447f7bfc-p48xw"
Jul 23 17:48:13.282668 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:48:13.282638    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-b9d590/sub-test-model-b9d590-kserve-9447f7bfc-p48xw"]
Jul 23 17:48:13.932164 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:48:13.932123    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-b9d590/sub-test-model-b9d590-kserve-9447f7bfc-p48xw" event={"ID":"821216c1-086f-4287-80e2-be36b4bdc07f","Type":"ContainerStarted","Data":"12479fe278928ffe4f13d8f23654dd4cc417b88e4f8110804f9bf5d10f6502e8"}
Jul 23 17:48:13.932559 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:48:13.932178    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-b9d590/sub-test-model-b9d590-kserve-9447f7bfc-p48xw" event={"ID":"821216c1-086f-4287-80e2-be36b4bdc07f","Type":"ContainerStarted","Data":"dfdeb33582ba8f020626448285e1fd02a69e3caecdee4a2fb2ca7c31cf93e08c"}
Jul 23 17:48:13.953970 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:48:13.953916    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-b9d590/sub-test-model-b9d590-kserve-9447f7bfc-p48xw" podStartSLOduration=1.953896876 podStartE2EDuration="1.953896876s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 17:48:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 17:48:13.950957404 +0000 UTC m=+2589.721242948" watchObservedRunningTime="2026-07-23 17:48:13.953896876 +0000 UTC m=+2589.724182396"
Jul 23 17:48:17.323865 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:48:17.323817    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-9c4020/auth-test-model-9c4020-kserve-6787fd5787-b2jhf" podUID="90d33a4a-220d-40a3-a344-6a6b139eb67d" containerName="main" containerID="cri-o://4f6dda3bf50f7aa9d83fe19d28b966acc89e7a13c313b4b1899cd0b8b5f4f92a" gracePeriod=45
Jul 23 17:48:17.571896 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:48:17.571873    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-9c4020/auth-test-model-9c4020-kserve-6787fd5787-b2jhf"
Jul 23 17:48:17.703597 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:48:17.703571    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/90d33a4a-220d-40a3-a344-6a6b139eb67d-dshm\") pod \"90d33a4a-220d-40a3-a344-6a6b139eb67d\" (UID: \"90d33a4a-220d-40a3-a344-6a6b139eb67d\") "
Jul 23 17:48:17.703754 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:48:17.703605    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/90d33a4a-220d-40a3-a344-6a6b139eb67d-home\") pod \"90d33a4a-220d-40a3-a344-6a6b139eb67d\" (UID: \"90d33a4a-220d-40a3-a344-6a6b139eb67d\") "
Jul 23 17:48:17.703754 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:48:17.703626    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/90d33a4a-220d-40a3-a344-6a6b139eb67d-model-cache\") pod \"90d33a4a-220d-40a3-a344-6a6b139eb67d\" (UID: \"90d33a4a-220d-40a3-a344-6a6b139eb67d\") "
Jul 23 17:48:17.703754 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:48:17.703672    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90d33a4a-220d-40a3-a344-6a6b139eb67d-tls-certs\") pod \"90d33a4a-220d-40a3-a344-6a6b139eb67d\" (UID: \"90d33a4a-220d-40a3-a344-6a6b139eb67d\") "
Jul 23 17:48:17.703754 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:48:17.703694    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90d33a4a-220d-40a3-a344-6a6b139eb67d-tmp-dir\") pod \"90d33a4a-220d-40a3-a344-6a6b139eb67d\" (UID: \"90d33a4a-220d-40a3-a344-6a6b139eb67d\") "
Jul 23 17:48:17.703754 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:48:17.703735    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2wlkk\" (UniqueName: \"kubernetes.io/projected/90d33a4a-220d-40a3-a344-6a6b139eb67d-kube-api-access-2wlkk\") pod \"90d33a4a-220d-40a3-a344-6a6b139eb67d\" (UID: \"90d33a4a-220d-40a3-a344-6a6b139eb67d\") "
Jul 23 17:48:17.704027 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:48:17.703875    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90d33a4a-220d-40a3-a344-6a6b139eb67d-home" (OuterVolumeSpecName: "home") pod "90d33a4a-220d-40a3-a344-6a6b139eb67d" (UID: "90d33a4a-220d-40a3-a344-6a6b139eb67d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 17:48:17.704027 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:48:17.703948    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90d33a4a-220d-40a3-a344-6a6b139eb67d-model-cache" (OuterVolumeSpecName: "model-cache") pod "90d33a4a-220d-40a3-a344-6a6b139eb67d" (UID: "90d33a4a-220d-40a3-a344-6a6b139eb67d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 17:48:17.704027 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:48:17.703996    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90d33a4a-220d-40a3-a344-6a6b139eb67d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "90d33a4a-220d-40a3-a344-6a6b139eb67d" (UID: "90d33a4a-220d-40a3-a344-6a6b139eb67d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 17:48:17.704177 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:48:17.704141    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/90d33a4a-220d-40a3-a344-6a6b139eb67d-home\") on node \"ip-10-0-129-216.ec2.internal\" DevicePath \"\""
Jul 23 17:48:17.704177 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:48:17.704161    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/90d33a4a-220d-40a3-a344-6a6b139eb67d-model-cache\") on node \"ip-10-0-129-216.ec2.internal\" DevicePath \"\""
Jul 23 17:48:17.704283 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:48:17.704176    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90d33a4a-220d-40a3-a344-6a6b139eb67d-tmp-dir\") on node \"ip-10-0-129-216.ec2.internal\" DevicePath \"\""
Jul 23 17:48:17.705841 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:48:17.705820    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90d33a4a-220d-40a3-a344-6a6b139eb67d-dshm" (OuterVolumeSpecName: "dshm") pod "90d33a4a-220d-40a3-a344-6a6b139eb67d" (UID: "90d33a4a-220d-40a3-a344-6a6b139eb67d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 17:48:17.705902 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:48:17.705884    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90d33a4a-220d-40a3-a344-6a6b139eb67d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "90d33a4a-220d-40a3-a344-6a6b139eb67d" (UID: "90d33a4a-220d-40a3-a344-6a6b139eb67d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 17:48:17.705979 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:48:17.705962    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90d33a4a-220d-40a3-a344-6a6b139eb67d-kube-api-access-2wlkk" (OuterVolumeSpecName: "kube-api-access-2wlkk") pod "90d33a4a-220d-40a3-a344-6a6b139eb67d" (UID: "90d33a4a-220d-40a3-a344-6a6b139eb67d"). InnerVolumeSpecName "kube-api-access-2wlkk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 17:48:17.804943 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:48:17.804915    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/90d33a4a-220d-40a3-a344-6a6b139eb67d-dshm\") on node \"ip-10-0-129-216.ec2.internal\" DevicePath \"\""
Jul 23 17:48:17.804943 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:48:17.804941    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90d33a4a-220d-40a3-a344-6a6b139eb67d-tls-certs\") on node \"ip-10-0-129-216.ec2.internal\" DevicePath \"\""
Jul 23 17:48:17.804943 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:48:17.804952    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2wlkk\" (UniqueName: \"kubernetes.io/projected/90d33a4a-220d-40a3-a344-6a6b139eb67d-kube-api-access-2wlkk\") on node \"ip-10-0-129-216.ec2.internal\" DevicePath \"\""
Jul 23 17:48:17.958037 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:48:17.957951    2577 generic.go:356] "Generic (PLEG): container finished" podID="90d33a4a-220d-40a3-a344-6a6b139eb67d" containerID="4f6dda3bf50f7aa9d83fe19d28b966acc89e7a13c313b4b1899cd0b8b5f4f92a" exitCode=0
Jul 23 17:48:17.958037 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:48:17.957999    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-9c4020/auth-test-model-9c4020-kserve-6787fd5787-b2jhf"
Jul 23 17:48:17.958231 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:48:17.958031    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-9c4020/auth-test-model-9c4020-kserve-6787fd5787-b2jhf" event={"ID":"90d33a4a-220d-40a3-a344-6a6b139eb67d","Type":"ContainerDied","Data":"4f6dda3bf50f7aa9d83fe19d28b966acc89e7a13c313b4b1899cd0b8b5f4f92a"}
Jul 23 17:48:17.958231 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:48:17.958075    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-9c4020/auth-test-model-9c4020-kserve-6787fd5787-b2jhf" event={"ID":"90d33a4a-220d-40a3-a344-6a6b139eb67d","Type":"ContainerDied","Data":"66dbddf9529473776bad932286372f2e9d7007bbedff39d58489633a5ce18769"}
Jul 23 17:48:17.958231 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:48:17.958093    2577 scope.go:117] "RemoveContainer" containerID="4f6dda3bf50f7aa9d83fe19d28b966acc89e7a13c313b4b1899cd0b8b5f4f92a"
Jul 23 17:48:17.967757 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:48:17.967741    2577 scope.go:117] "RemoveContainer" containerID="4f6dda3bf50f7aa9d83fe19d28b966acc89e7a13c313b4b1899cd0b8b5f4f92a"
Jul 23 17:48:17.968004 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:48:17.967989    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f6dda3bf50f7aa9d83fe19d28b966acc89e7a13c313b4b1899cd0b8b5f4f92a\": container with ID starting with 4f6dda3bf50f7aa9d83fe19d28b966acc89e7a13c313b4b1899cd0b8b5f4f92a not found: ID does not exist" containerID="4f6dda3bf50f7aa9d83fe19d28b966acc89e7a13c313b4b1899cd0b8b5f4f92a"
Jul 23 17:48:17.968044 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:48:17.968011    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f6dda3bf50f7aa9d83fe19d28b966acc89e7a13c313b4b1899cd0b8b5f4f92a"} err="failed to get container status \"4f6dda3bf50f7aa9d83fe19d28b966acc89e7a13c313b4b1899cd0b8b5f4f92a\": rpc error: code = NotFound desc = could not find container \"4f6dda3bf50f7aa9d83fe19d28b966acc89e7a13c313b4b1899cd0b8b5f4f92a\": container with ID starting with 4f6dda3bf50f7aa9d83fe19d28b966acc89e7a13c313b4b1899cd0b8b5f4f92a not found: ID does not exist"
Jul 23 17:48:17.979652 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:48:17.979629    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-9c4020/auth-test-model-9c4020-kserve-6787fd5787-b2jhf"]
Jul 23 17:48:17.985328 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:48:17.985304    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-9c4020/auth-test-model-9c4020-kserve-6787fd5787-b2jhf"]
Jul 23 17:48:18.766473 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:48:18.766442    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="90d33a4a-220d-40a3-a344-6a6b139eb67d" path="/var/lib/kubelet/pods/90d33a4a-220d-40a3-a344-6a6b139eb67d/volumes"
Jul 23 17:48:23.119399 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:48:23.119368    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-b9d590/sub-test-model-b9d590-kserve-9447f7bfc-p48xw"
Jul 23 17:48:23.119791 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:48:23.119418    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-b9d590/sub-test-model-b9d590-kserve-9447f7bfc-p48xw"
Jul 23 17:48:23.132035 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:48:23.131998    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-b9d590/sub-test-model-b9d590-kserve-9447f7bfc-p48xw"
Jul 23 17:48:24.005920 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:48:24.005888    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-b9d590/sub-test-model-b9d590-kserve-9447f7bfc-p48xw"
Jul 23 17:48:41.899862 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:48:41.899779    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:49:05.082200 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:49:05.082161    2577 scope.go:117] "RemoveContainer" containerID="32ce7369805e742fbccb616125b51558c1719490356a3be3c7c3171375cdf42a"
Jul 23 17:49:26.829806 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:49:26.829767    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-b9d590/sub-test-model-b9d590-kserve-9447f7bfc-p48xw"]
Jul 23 17:49:26.854486 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:49:26.854448    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:49:27.830214 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:49:27.830169    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:49:39.234812 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:49:39.234769    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-b9d590/rate-test-model-b9d590-kserve-679779687f-jccfk"]
Jul 23 17:49:39.235443 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:49:39.235244    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="90d33a4a-220d-40a3-a344-6a6b139eb67d" containerName="main"
Jul 23 17:49:39.235443 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:49:39.235271    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="90d33a4a-220d-40a3-a344-6a6b139eb67d" containerName="main"
Jul 23 17:49:39.235443 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:49:39.235346    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="90d33a4a-220d-40a3-a344-6a6b139eb67d" containerName="main"
Jul 23 17:49:39.239006 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:49:39.238984    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-b9d590/rate-test-model-b9d590-kserve-679779687f-jccfk"
Jul 23 17:49:39.242031 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:49:39.242010    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-b9d590\"/\"rate-test-model-b9d590-kserve-self-signed-certs\""
Jul 23 17:49:39.259382 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:49:39.259361    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/229d3b28-4807-4e27-8a48-74b2864154fc-dshm\") pod \"rate-test-model-b9d590-kserve-679779687f-jccfk\" (UID: \"229d3b28-4807-4e27-8a48-74b2864154fc\") " pod="ai-tenant-e2e-shared-a-b9d590/rate-test-model-b9d590-kserve-679779687f-jccfk"
Jul 23 17:49:39.259486 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:49:39.259400    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/229d3b28-4807-4e27-8a48-74b2864154fc-home\") pod \"rate-test-model-b9d590-kserve-679779687f-jccfk\" (UID: \"229d3b28-4807-4e27-8a48-74b2864154fc\") " pod="ai-tenant-e2e-shared-a-b9d590/rate-test-model-b9d590-kserve-679779687f-jccfk"
Jul 23 17:49:39.259486 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:49:39.259450    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/229d3b28-4807-4e27-8a48-74b2864154fc-tmp-dir\") pod \"rate-test-model-b9d590-kserve-679779687f-jccfk\" (UID: \"229d3b28-4807-4e27-8a48-74b2864154fc\") " pod="ai-tenant-e2e-shared-a-b9d590/rate-test-model-b9d590-kserve-679779687f-jccfk"
Jul 23 17:49:39.259486 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:49:39.259474    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8zb2\" (UniqueName: \"kubernetes.io/projected/229d3b28-4807-4e27-8a48-74b2864154fc-kube-api-access-l8zb2\") pod \"rate-test-model-b9d590-kserve-679779687f-jccfk\" (UID: \"229d3b28-4807-4e27-8a48-74b2864154fc\") " pod="ai-tenant-e2e-shared-a-b9d590/rate-test-model-b9d590-kserve-679779687f-jccfk"
Jul 23 17:49:39.259628 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:49:39.259535    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/229d3b28-4807-4e27-8a48-74b2864154fc-tls-certs\") pod \"rate-test-model-b9d590-kserve-679779687f-jccfk\" (UID: \"229d3b28-4807-4e27-8a48-74b2864154fc\") " pod="ai-tenant-e2e-shared-a-b9d590/rate-test-model-b9d590-kserve-679779687f-jccfk"
Jul 23 17:49:39.259628 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:49:39.259575    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/229d3b28-4807-4e27-8a48-74b2864154fc-model-cache\") pod \"rate-test-model-b9d590-kserve-679779687f-jccfk\" (UID: \"229d3b28-4807-4e27-8a48-74b2864154fc\") " pod="ai-tenant-e2e-shared-a-b9d590/rate-test-model-b9d590-kserve-679779687f-jccfk"
Jul 23 17:49:39.263193 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:49:39.263175    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-b9d590/rate-test-model-b9d590-kserve-679779687f-jccfk"]
Jul 23 17:49:39.321348 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:49:39.321317    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:49:39.360515 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:49:39.360490    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/229d3b28-4807-4e27-8a48-74b2864154fc-dshm\") pod \"rate-test-model-b9d590-kserve-679779687f-jccfk\" (UID: \"229d3b28-4807-4e27-8a48-74b2864154fc\") " pod="ai-tenant-e2e-shared-a-b9d590/rate-test-model-b9d590-kserve-679779687f-jccfk"
Jul 23 17:49:39.360656 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:49:39.360527    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/229d3b28-4807-4e27-8a48-74b2864154fc-home\") pod \"rate-test-model-b9d590-kserve-679779687f-jccfk\" (UID: \"229d3b28-4807-4e27-8a48-74b2864154fc\") " pod="ai-tenant-e2e-shared-a-b9d590/rate-test-model-b9d590-kserve-679779687f-jccfk"
Jul 23 17:49:39.360701 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:49:39.360657    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/229d3b28-4807-4e27-8a48-74b2864154fc-tmp-dir\") pod \"rate-test-model-b9d590-kserve-679779687f-jccfk\" (UID: \"229d3b28-4807-4e27-8a48-74b2864154fc\") " pod="ai-tenant-e2e-shared-a-b9d590/rate-test-model-b9d590-kserve-679779687f-jccfk"
Jul 23 17:49:39.360747 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:49:39.360695    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8zb2\" (UniqueName: \"kubernetes.io/projected/229d3b28-4807-4e27-8a48-74b2864154fc-kube-api-access-l8zb2\") pod \"rate-test-model-b9d590-kserve-679779687f-jccfk\" (UID: \"229d3b28-4807-4e27-8a48-74b2864154fc\") " pod="ai-tenant-e2e-shared-a-b9d590/rate-test-model-b9d590-kserve-679779687f-jccfk"
Jul 23 17:49:39.360787 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:49:39.360748    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/229d3b28-4807-4e27-8a48-74b2864154fc-tls-certs\") pod \"rate-test-model-b9d590-kserve-679779687f-jccfk\" (UID: \"229d3b28-4807-4e27-8a48-74b2864154fc\") " pod="ai-tenant-e2e-shared-a-b9d590/rate-test-model-b9d590-kserve-679779687f-jccfk"
Jul 23 17:49:39.360787 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:49:39.360782    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/229d3b28-4807-4e27-8a48-74b2864154fc-model-cache\") pod \"rate-test-model-b9d590-kserve-679779687f-jccfk\" (UID: \"229d3b28-4807-4e27-8a48-74b2864154fc\") " pod="ai-tenant-e2e-shared-a-b9d590/rate-test-model-b9d590-kserve-679779687f-jccfk"
Jul 23 17:49:39.360938 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:49:39.360895    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/229d3b28-4807-4e27-8a48-74b2864154fc-home\") pod \"rate-test-model-b9d590-kserve-679779687f-jccfk\" (UID: \"229d3b28-4807-4e27-8a48-74b2864154fc\") " pod="ai-tenant-e2e-shared-a-b9d590/rate-test-model-b9d590-kserve-679779687f-jccfk"
Jul 23 17:49:39.361059 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:49:39.360960    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/229d3b28-4807-4e27-8a48-74b2864154fc-tmp-dir\") pod \"rate-test-model-b9d590-kserve-679779687f-jccfk\" (UID: \"229d3b28-4807-4e27-8a48-74b2864154fc\") " pod="ai-tenant-e2e-shared-a-b9d590/rate-test-model-b9d590-kserve-679779687f-jccfk"
Jul 23 17:49:39.361113 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:49:39.361098    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/229d3b28-4807-4e27-8a48-74b2864154fc-model-cache\") pod \"rate-test-model-b9d590-kserve-679779687f-jccfk\" (UID: \"229d3b28-4807-4e27-8a48-74b2864154fc\") " pod="ai-tenant-e2e-shared-a-b9d590/rate-test-model-b9d590-kserve-679779687f-jccfk"
Jul 23 17:49:39.362775 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:49:39.362749    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/229d3b28-4807-4e27-8a48-74b2864154fc-dshm\") pod \"rate-test-model-b9d590-kserve-679779687f-jccfk\" (UID: \"229d3b28-4807-4e27-8a48-74b2864154fc\") " pod="ai-tenant-e2e-shared-a-b9d590/rate-test-model-b9d590-kserve-679779687f-jccfk"
Jul 23 17:49:39.363219 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:49:39.363195    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/229d3b28-4807-4e27-8a48-74b2864154fc-tls-certs\") pod \"rate-test-model-b9d590-kserve-679779687f-jccfk\" (UID: \"229d3b28-4807-4e27-8a48-74b2864154fc\") " pod="ai-tenant-e2e-shared-a-b9d590/rate-test-model-b9d590-kserve-679779687f-jccfk"
Jul 23 17:49:39.375613 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:49:39.375590    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8zb2\" (UniqueName: \"kubernetes.io/projected/229d3b28-4807-4e27-8a48-74b2864154fc-kube-api-access-l8zb2\") pod \"rate-test-model-b9d590-kserve-679779687f-jccfk\" (UID: \"229d3b28-4807-4e27-8a48-74b2864154fc\") " pod="ai-tenant-e2e-shared-a-b9d590/rate-test-model-b9d590-kserve-679779687f-jccfk"
Jul 23 17:49:39.549112 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:49:39.549033    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-b9d590/rate-test-model-b9d590-kserve-679779687f-jccfk"
Jul 23 17:49:39.691266 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:49:39.691232    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-b9d590/rate-test-model-b9d590-kserve-679779687f-jccfk"]
Jul 23 17:49:39.692051 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:49:39.692016    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod229d3b28_4807_4e27_8a48_74b2864154fc.slice/crio-463379c2f9616d93322f240a951276c511015edfefd42082b5675661506bef60 WatchSource:0}: Error finding container 463379c2f9616d93322f240a951276c511015edfefd42082b5675661506bef60: Status 404 returned error can't find the container with id 463379c2f9616d93322f240a951276c511015edfefd42082b5675661506bef60
Jul 23 17:49:40.445620 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:49:40.445577    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-b9d590/rate-test-model-b9d590-kserve-679779687f-jccfk" event={"ID":"229d3b28-4807-4e27-8a48-74b2864154fc","Type":"ContainerStarted","Data":"ee692dd0fbf98fdb6c20dc151e0eef04cc900c7ea0fe4df521bbb8b93e42545d"}
Jul 23 17:49:40.446104 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:49:40.445630    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-b9d590/rate-test-model-b9d590-kserve-679779687f-jccfk" event={"ID":"229d3b28-4807-4e27-8a48-74b2864154fc","Type":"ContainerStarted","Data":"463379c2f9616d93322f240a951276c511015edfefd42082b5675661506bef60"}
Jul 23 17:49:40.463808 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:49:40.463754    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-b9d590/rate-test-model-b9d590-kserve-679779687f-jccfk" podStartSLOduration=1.463739119 podStartE2EDuration="1.463739119s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 17:49:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 17:49:40.461569358 +0000 UTC m=+2676.231854893" watchObservedRunningTime="2026-07-23 17:49:40.463739119 +0000 UTC m=+2676.234024637"
Jul 23 17:49:41.850819 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:49:41.850780    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-b9d590/sub-test-model-b9d590-kserve-9447f7bfc-p48xw" podUID="821216c1-086f-4287-80e2-be36b4bdc07f" containerName="main" containerID="cri-o://12479fe278928ffe4f13d8f23654dd4cc417b88e4f8110804f9bf5d10f6502e8" gracePeriod=45
Jul 23 17:49:42.116127 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:49:42.116069    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-b9d590/sub-test-model-b9d590-kserve-9447f7bfc-p48xw"
Jul 23 17:49:42.181670 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:49:42.181641    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/821216c1-086f-4287-80e2-be36b4bdc07f-dshm\") pod \"821216c1-086f-4287-80e2-be36b4bdc07f\" (UID: \"821216c1-086f-4287-80e2-be36b4bdc07f\") "
Jul 23 17:49:42.181840 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:49:42.181699    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/821216c1-086f-4287-80e2-be36b4bdc07f-home\") pod \"821216c1-086f-4287-80e2-be36b4bdc07f\" (UID: \"821216c1-086f-4287-80e2-be36b4bdc07f\") "
Jul 23 17:49:42.181840 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:49:42.181719    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/821216c1-086f-4287-80e2-be36b4bdc07f-tls-certs\") pod \"821216c1-086f-4287-80e2-be36b4bdc07f\" (UID: \"821216c1-086f-4287-80e2-be36b4bdc07f\") "
Jul 23 17:49:42.181840 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:49:42.181743    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2vjsd\" (UniqueName: \"kubernetes.io/projected/821216c1-086f-4287-80e2-be36b4bdc07f-kube-api-access-2vjsd\") pod \"821216c1-086f-4287-80e2-be36b4bdc07f\" (UID: \"821216c1-086f-4287-80e2-be36b4bdc07f\") "
Jul 23 17:49:42.181840 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:49:42.181790    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/821216c1-086f-4287-80e2-be36b4bdc07f-tmp-dir\") pod \"821216c1-086f-4287-80e2-be36b4bdc07f\" (UID: \"821216c1-086f-4287-80e2-be36b4bdc07f\") "
Jul 23 17:49:42.181840 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:49:42.181826    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/821216c1-086f-4287-80e2-be36b4bdc07f-model-cache\") pod \"821216c1-086f-4287-80e2-be36b4bdc07f\" (UID: \"821216c1-086f-4287-80e2-be36b4bdc07f\") "
Jul 23 17:49:42.182124 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:49:42.181916    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/821216c1-086f-4287-80e2-be36b4bdc07f-home" (OuterVolumeSpecName: "home") pod "821216c1-086f-4287-80e2-be36b4bdc07f" (UID: "821216c1-086f-4287-80e2-be36b4bdc07f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 17:49:42.182124 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:49:42.182108    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/821216c1-086f-4287-80e2-be36b4bdc07f-home\") on node \"ip-10-0-129-216.ec2.internal\" DevicePath \"\""
Jul 23 17:49:42.182221 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:49:42.182118    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/821216c1-086f-4287-80e2-be36b4bdc07f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "821216c1-086f-4287-80e2-be36b4bdc07f" (UID: "821216c1-086f-4287-80e2-be36b4bdc07f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 17:49:42.182221 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:49:42.182130    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/821216c1-086f-4287-80e2-be36b4bdc07f-model-cache" (OuterVolumeSpecName: "model-cache") pod "821216c1-086f-4287-80e2-be36b4bdc07f" (UID: "821216c1-086f-4287-80e2-be36b4bdc07f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 17:49:42.183900 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:49:42.183872    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/821216c1-086f-4287-80e2-be36b4bdc07f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "821216c1-086f-4287-80e2-be36b4bdc07f" (UID: "821216c1-086f-4287-80e2-be36b4bdc07f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 17:49:42.183900 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:49:42.183890    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/821216c1-086f-4287-80e2-be36b4bdc07f-dshm" (OuterVolumeSpecName: "dshm") pod "821216c1-086f-4287-80e2-be36b4bdc07f" (UID: "821216c1-086f-4287-80e2-be36b4bdc07f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 17:49:42.184046 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:49:42.183958    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/821216c1-086f-4287-80e2-be36b4bdc07f-kube-api-access-2vjsd" (OuterVolumeSpecName: "kube-api-access-2vjsd") pod "821216c1-086f-4287-80e2-be36b4bdc07f" (UID: "821216c1-086f-4287-80e2-be36b4bdc07f"). InnerVolumeSpecName "kube-api-access-2vjsd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 17:49:42.282654 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:49:42.282620    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/821216c1-086f-4287-80e2-be36b4bdc07f-tmp-dir\") on node \"ip-10-0-129-216.ec2.internal\" DevicePath \"\""
Jul 23 17:49:42.282654 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:49:42.282650    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/821216c1-086f-4287-80e2-be36b4bdc07f-model-cache\") on node \"ip-10-0-129-216.ec2.internal\" DevicePath \"\""
Jul 23 17:49:42.282654 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:49:42.282661    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/821216c1-086f-4287-80e2-be36b4bdc07f-dshm\") on node \"ip-10-0-129-216.ec2.internal\" DevicePath \"\""
Jul 23 17:49:42.282879 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:49:42.282671    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/821216c1-086f-4287-80e2-be36b4bdc07f-tls-certs\") on node \"ip-10-0-129-216.ec2.internal\" DevicePath \"\""
Jul 23 17:49:42.282879 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:49:42.282680    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2vjsd\" (UniqueName: \"kubernetes.io/projected/821216c1-086f-4287-80e2-be36b4bdc07f-kube-api-access-2vjsd\") on node \"ip-10-0-129-216.ec2.internal\" DevicePath \"\""
Jul 23 17:49:42.458864 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:49:42.458829    2577 generic.go:356] "Generic (PLEG): container finished" podID="821216c1-086f-4287-80e2-be36b4bdc07f" containerID="12479fe278928ffe4f13d8f23654dd4cc417b88e4f8110804f9bf5d10f6502e8" exitCode=0
Jul 23 17:49:42.459057 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:49:42.458879    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-b9d590/sub-test-model-b9d590-kserve-9447f7bfc-p48xw"
Jul 23 17:49:42.459057 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:49:42.458920    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-b9d590/sub-test-model-b9d590-kserve-9447f7bfc-p48xw" event={"ID":"821216c1-086f-4287-80e2-be36b4bdc07f","Type":"ContainerDied","Data":"12479fe278928ffe4f13d8f23654dd4cc417b88e4f8110804f9bf5d10f6502e8"}
Jul 23 17:49:42.459057 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:49:42.458977    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-b9d590/sub-test-model-b9d590-kserve-9447f7bfc-p48xw" event={"ID":"821216c1-086f-4287-80e2-be36b4bdc07f","Type":"ContainerDied","Data":"dfdeb33582ba8f020626448285e1fd02a69e3caecdee4a2fb2ca7c31cf93e08c"}
Jul 23 17:49:42.459057 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:49:42.459002    2577 scope.go:117] "RemoveContainer" containerID="12479fe278928ffe4f13d8f23654dd4cc417b88e4f8110804f9bf5d10f6502e8"
Jul 23 17:49:42.468488 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:49:42.468471    2577 scope.go:117] "RemoveContainer" containerID="12479fe278928ffe4f13d8f23654dd4cc417b88e4f8110804f9bf5d10f6502e8"
Jul 23 17:49:42.468721 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:49:42.468703    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12479fe278928ffe4f13d8f23654dd4cc417b88e4f8110804f9bf5d10f6502e8\": container with ID starting with 12479fe278928ffe4f13d8f23654dd4cc417b88e4f8110804f9bf5d10f6502e8 not found: ID does not exist" containerID="12479fe278928ffe4f13d8f23654dd4cc417b88e4f8110804f9bf5d10f6502e8"
Jul 23 17:49:42.468792 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:49:42.468727    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12479fe278928ffe4f13d8f23654dd4cc417b88e4f8110804f9bf5d10f6502e8"} err="failed to get container status \"12479fe278928ffe4f13d8f23654dd4cc417b88e4f8110804f9bf5d10f6502e8\": rpc error: code = NotFound desc = could not find container \"12479fe278928ffe4f13d8f23654dd4cc417b88e4f8110804f9bf5d10f6502e8\": container with ID starting with 12479fe278928ffe4f13d8f23654dd4cc417b88e4f8110804f9bf5d10f6502e8 not found: ID does not exist"
Jul 23 17:49:42.482228 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:49:42.482204    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-b9d590/sub-test-model-b9d590-kserve-9447f7bfc-p48xw"]
Jul 23 17:49:42.486964 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:49:42.486940    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-b9d590/sub-test-model-b9d590-kserve-9447f7bfc-p48xw"]
Jul 23 17:49:42.766739 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:49:42.766659    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="821216c1-086f-4287-80e2-be36b4bdc07f" path="/var/lib/kubelet/pods/821216c1-086f-4287-80e2-be36b4bdc07f/volumes"
Jul 23 17:49:45.784148 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:49:45.784111    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:49:45.885397 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:49:45.885367    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:49:46.265073 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:49:46.265041    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:49:46.368444 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:49:46.368404    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:49:49.549698 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:49:49.549663    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-b9d590/rate-test-model-b9d590-kserve-679779687f-jccfk"
Jul 23 17:49:49.550109 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:49:49.549728    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-b9d590/rate-test-model-b9d590-kserve-679779687f-jccfk"
Jul 23 17:49:49.562506 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:49:49.562483    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-b9d590/rate-test-model-b9d590-kserve-679779687f-jccfk"
Jul 23 17:49:50.524598 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:49:50.524563    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-b9d590/rate-test-model-b9d590-kserve-679779687f-jccfk"
Jul 23 17:50:03.824008 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:50:03.823921    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:50:05.112616 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:50:05.112585    2577 scope.go:117] "RemoveContainer" containerID="f09e2f0a3323914316b19f80f8920c77d7b7ac3ab36541f85ef23d0a22d73ea3"
Jul 23 17:50:13.624480 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:50:13.624446    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:50:13.725513 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:50:13.725479    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:50:23.577136 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:50:23.577104    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:50:23.677647 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:50:23.677615    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:50:26.471517 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:50:26.471480    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:50:40.931925 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:50:40.931890    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-b9d590/rate-test-model-b9d590-kserve-679779687f-jccfk"]
Jul 23 17:50:40.956071 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:50:40.956035    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:50:41.899539 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:50:41.899505    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:50:42.865413 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:50:42.865378    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7328-140135558958768"]
Jul 23 17:50:42.865824 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:50:42.865811    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="821216c1-086f-4287-80e2-be36b4bdc07f" containerName="main"
Jul 23 17:50:42.865824 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:50:42.865825    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="821216c1-086f-4287-80e2-be36b4bdc07f" containerName="main"
Jul 23 17:50:42.865900 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:50:42.865894    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="821216c1-086f-4287-80e2-be36b4bdc07f" containerName="main"
Jul 23 17:50:42.868942 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:50:42.868926    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7328-140135558958768"
Jul 23 17:50:42.871580 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:50:42.871564    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-qpwfj\""
Jul 23 17:50:42.878466 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:50:42.878439    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7328-140135558958768"]
Jul 23 17:50:42.935013 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:50:42.934970    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srv9r\" (UniqueName: \"kubernetes.io/projected/f451881e-42ba-44aa-9cf8-3f182af4f66e-kube-api-access-srv9r\") pod \"test-curl-7328-140135558958768\" (UID: \"f451881e-42ba-44aa-9cf8-3f182af4f66e\") " pod="opendatahub/test-curl-7328-140135558958768"
Jul 23 17:50:43.035917 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:50:43.035879    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srv9r\" (UniqueName: \"kubernetes.io/projected/f451881e-42ba-44aa-9cf8-3f182af4f66e-kube-api-access-srv9r\") pod \"test-curl-7328-140135558958768\" (UID: \"f451881e-42ba-44aa-9cf8-3f182af4f66e\") " pod="opendatahub/test-curl-7328-140135558958768"
Jul 23 17:50:43.049185 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:50:43.049157    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-srv9r\" (UniqueName: \"kubernetes.io/projected/f451881e-42ba-44aa-9cf8-3f182af4f66e-kube-api-access-srv9r\") pod \"test-curl-7328-140135558958768\" (UID: \"f451881e-42ba-44aa-9cf8-3f182af4f66e\") " pod="opendatahub/test-curl-7328-140135558958768"
Jul 23 17:50:43.179162 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:50:43.179078    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7328-140135558958768"
Jul 23 17:50:43.319182 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:50:43.319157    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7328-140135558958768"]
Jul 23 17:50:43.320657 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:50:43.320625    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf451881e_42ba_44aa_9cf8_3f182af4f66e.slice/crio-b994ef43fc7ca6f329e460960c886fbe900723eb3ba490b2ae05166d844c8b3b WatchSource:0}: Error finding container b994ef43fc7ca6f329e460960c886fbe900723eb3ba490b2ae05166d844c8b3b: Status 404 returned error can't find the container with id b994ef43fc7ca6f329e460960c886fbe900723eb3ba490b2ae05166d844c8b3b
Jul 23 17:50:43.322014 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:50:43.321993    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 23 17:50:43.835533 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:50:43.835470    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7328-140135558958768" event={"ID":"f451881e-42ba-44aa-9cf8-3f182af4f66e","Type":"ContainerStarted","Data":"b994ef43fc7ca6f329e460960c886fbe900723eb3ba490b2ae05166d844c8b3b"}
Jul 23 17:50:44.843306 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:50:44.843270    2577 generic.go:356] "Generic (PLEG): container finished" podID="f451881e-42ba-44aa-9cf8-3f182af4f66e" containerID="64c121762593fba4516d5cd4326d1c47aad19b24578f101f0912fa7a66134301" exitCode=0
Jul 23 17:50:44.843779 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:50:44.843347    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7328-140135558958768" event={"ID":"f451881e-42ba-44aa-9cf8-3f182af4f66e","Type":"ContainerDied","Data":"64c121762593fba4516d5cd4326d1c47aad19b24578f101f0912fa7a66134301"}
Jul 23 17:50:45.982615 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:50:45.982592    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7328-140135558958768"
Jul 23 17:50:46.005914 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:50:46.005889    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7328-140135558958768_f451881e-42ba-44aa-9cf8-3f182af4f66e/test-curl-7328-140135558958768/0.log"
Jul 23 17:50:46.032718 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:50:46.032686    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7328-140135558958768"]
Jul 23 17:50:46.036754 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:50:46.036725    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7328-140135558958768"]
Jul 23 17:50:46.063614 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:50:46.063590    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-srv9r\" (UniqueName: \"kubernetes.io/projected/f451881e-42ba-44aa-9cf8-3f182af4f66e-kube-api-access-srv9r\") pod \"f451881e-42ba-44aa-9cf8-3f182af4f66e\" (UID: \"f451881e-42ba-44aa-9cf8-3f182af4f66e\") "
Jul 23 17:50:46.065594 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:50:46.065568    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f451881e-42ba-44aa-9cf8-3f182af4f66e-kube-api-access-srv9r" (OuterVolumeSpecName: "kube-api-access-srv9r") pod "f451881e-42ba-44aa-9cf8-3f182af4f66e" (UID: "f451881e-42ba-44aa-9cf8-3f182af4f66e"). InnerVolumeSpecName "kube-api-access-srv9r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 17:50:46.164892 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:50:46.164812    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-srv9r\" (UniqueName: \"kubernetes.io/projected/f451881e-42ba-44aa-9cf8-3f182af4f66e-kube-api-access-srv9r\") on node \"ip-10-0-129-216.ec2.internal\" DevicePath \"\""
Jul 23 17:50:46.165105 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:50:46.165017    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7328-140135561176112"]
Jul 23 17:50:46.165461 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:50:46.165446    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f451881e-42ba-44aa-9cf8-3f182af4f66e" containerName="test-curl-7328-140135558958768"
Jul 23 17:50:46.165504 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:50:46.165463    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="f451881e-42ba-44aa-9cf8-3f182af4f66e" containerName="test-curl-7328-140135558958768"
Jul 23 17:50:46.165547 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:50:46.165538    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="f451881e-42ba-44aa-9cf8-3f182af4f66e" containerName="test-curl-7328-140135558958768"
Jul 23 17:50:46.171285 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:50:46.171246    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7328-140135561176112"
Jul 23 17:50:46.176913 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:50:46.176884    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7328-140135561176112"]
Jul 23 17:50:46.265843 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:50:46.265809    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbfz5\" (UniqueName: \"kubernetes.io/projected/74bea808-c60b-4b6c-aa38-31420cf780f1-kube-api-access-vbfz5\") pod \"test-curl-7328-140135561176112\" (UID: \"74bea808-c60b-4b6c-aa38-31420cf780f1\") " pod="opendatahub/test-curl-7328-140135561176112"
Jul 23 17:50:46.367317 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:50:46.367282    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbfz5\" (UniqueName: \"kubernetes.io/projected/74bea808-c60b-4b6c-aa38-31420cf780f1-kube-api-access-vbfz5\") pod \"test-curl-7328-140135561176112\" (UID: \"74bea808-c60b-4b6c-aa38-31420cf780f1\") " pod="opendatahub/test-curl-7328-140135561176112"
Jul 23 17:50:46.375787 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:50:46.375758    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbfz5\" (UniqueName: \"kubernetes.io/projected/74bea808-c60b-4b6c-aa38-31420cf780f1-kube-api-access-vbfz5\") pod \"test-curl-7328-140135561176112\" (UID: \"74bea808-c60b-4b6c-aa38-31420cf780f1\") " pod="opendatahub/test-curl-7328-140135561176112"
Jul 23 17:50:46.481967 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:50:46.481940    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7328-140135561176112"
Jul 23 17:50:46.619152 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:50:46.619125    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7328-140135561176112"]
Jul 23 17:50:46.620688 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:50:46.620662    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod74bea808_c60b_4b6c_aa38_31420cf780f1.slice/crio-c993bf1b9acaba496a38c2b4ba451f2d4e1e62d8721c9d70cb0c4aed711ae240 WatchSource:0}: Error finding container c993bf1b9acaba496a38c2b4ba451f2d4e1e62d8721c9d70cb0c4aed711ae240: Status 404 returned error can't find the container with id c993bf1b9acaba496a38c2b4ba451f2d4e1e62d8721c9d70cb0c4aed711ae240
Jul 23 17:50:46.766703 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:50:46.766626    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f451881e-42ba-44aa-9cf8-3f182af4f66e" path="/var/lib/kubelet/pods/f451881e-42ba-44aa-9cf8-3f182af4f66e/volumes"
Jul 23 17:50:46.856832 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:50:46.856796    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7328-140135561176112" event={"ID":"74bea808-c60b-4b6c-aa38-31420cf780f1","Type":"ContainerStarted","Data":"c993bf1b9acaba496a38c2b4ba451f2d4e1e62d8721c9d70cb0c4aed711ae240"}
Jul 23 17:50:46.858105 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:50:46.858084    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7328-140135558958768"
Jul 23 17:50:46.858214 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:50:46.858084    2577 scope.go:117] "RemoveContainer" containerID="64c121762593fba4516d5cd4326d1c47aad19b24578f101f0912fa7a66134301"
Jul 23 17:50:47.865596 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:50:47.865557    2577 generic.go:356] "Generic (PLEG): container finished" podID="74bea808-c60b-4b6c-aa38-31420cf780f1" containerID="1b1be357ecc3584e7e097416c52938dd797ae976478146d68e64a204c27415c0" exitCode=0
Jul 23 17:50:47.865983 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:50:47.865633    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7328-140135561176112" event={"ID":"74bea808-c60b-4b6c-aa38-31420cf780f1","Type":"ContainerDied","Data":"1b1be357ecc3584e7e097416c52938dd797ae976478146d68e64a204c27415c0"}
Jul 23 17:50:49.003533 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:50:49.003510    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7328-140135561176112"
Jul 23 17:50:49.025833 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:50:49.025806    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7328-140135561176112_74bea808-c60b-4b6c-aa38-31420cf780f1/test-curl-7328-140135561176112/0.log"
Jul 23 17:50:49.055249 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:50:49.055212    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7328-140135561176112"]
Jul 23 17:50:49.057603 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:50:49.057583    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7328-140135561176112"]
Jul 23 17:50:49.097597 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:50:49.097565    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vbfz5\" (UniqueName: \"kubernetes.io/projected/74bea808-c60b-4b6c-aa38-31420cf780f1-kube-api-access-vbfz5\") pod \"74bea808-c60b-4b6c-aa38-31420cf780f1\" (UID: \"74bea808-c60b-4b6c-aa38-31420cf780f1\") "
Jul 23 17:50:49.099620 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:50:49.099589    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74bea808-c60b-4b6c-aa38-31420cf780f1-kube-api-access-vbfz5" (OuterVolumeSpecName: "kube-api-access-vbfz5") pod "74bea808-c60b-4b6c-aa38-31420cf780f1" (UID: "74bea808-c60b-4b6c-aa38-31420cf780f1"). InnerVolumeSpecName "kube-api-access-vbfz5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 17:50:49.198492 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:50:49.198462    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vbfz5\" (UniqueName: \"kubernetes.io/projected/74bea808-c60b-4b6c-aa38-31420cf780f1-kube-api-access-vbfz5\") on node \"ip-10-0-129-216.ec2.internal\" DevicePath \"\""
Jul 23 17:50:49.230001 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:50:49.229971    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7328-140135561016928"]
Jul 23 17:50:49.230451 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:50:49.230434    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="74bea808-c60b-4b6c-aa38-31420cf780f1" containerName="test-curl-7328-140135561176112"
Jul 23 17:50:49.230451 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:50:49.230451    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="74bea808-c60b-4b6c-aa38-31420cf780f1" containerName="test-curl-7328-140135561176112"
Jul 23 17:50:49.230584 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:50:49.230527    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="74bea808-c60b-4b6c-aa38-31420cf780f1" containerName="test-curl-7328-140135561176112"
Jul 23 17:50:49.233549 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:50:49.233530    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7328-140135561016928"
Jul 23 17:50:49.246548 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:50:49.246527    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7328-140135561016928"]
Jul 23 17:50:49.299884 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:50:49.299860    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7jp6\" (UniqueName: \"kubernetes.io/projected/795a84aa-326a-4290-961e-9af262c37f43-kube-api-access-r7jp6\") pod \"test-curl-7328-140135561016928\" (UID: \"795a84aa-326a-4290-961e-9af262c37f43\") " pod="opendatahub/test-curl-7328-140135561016928"
Jul 23 17:50:49.400329 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:50:49.400302    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7jp6\" (UniqueName: \"kubernetes.io/projected/795a84aa-326a-4290-961e-9af262c37f43-kube-api-access-r7jp6\") pod \"test-curl-7328-140135561016928\" (UID: \"795a84aa-326a-4290-961e-9af262c37f43\") " pod="opendatahub/test-curl-7328-140135561016928"
Jul 23 17:50:49.411711 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:50:49.411678    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7jp6\" (UniqueName: \"kubernetes.io/projected/795a84aa-326a-4290-961e-9af262c37f43-kube-api-access-r7jp6\") pod \"test-curl-7328-140135561016928\" (UID: \"795a84aa-326a-4290-961e-9af262c37f43\") " pod="opendatahub/test-curl-7328-140135561016928"
Jul 23 17:50:49.542924 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:50:49.542848    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7328-140135561016928"
Jul 23 17:50:49.685671 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:50:49.685647    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7328-140135561016928"]
Jul 23 17:50:49.687202 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:50:49.687171    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod795a84aa_326a_4290_961e_9af262c37f43.slice/crio-cf3fbe64fd9e27494baa2ef10745d2a3734cf716f516304369ce6e7148f5ee39 WatchSource:0}: Error finding container cf3fbe64fd9e27494baa2ef10745d2a3734cf716f516304369ce6e7148f5ee39: Status 404 returned error can't find the container with id cf3fbe64fd9e27494baa2ef10745d2a3734cf716f516304369ce6e7148f5ee39
Jul 23 17:50:49.878004 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:50:49.877921    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7328-140135561176112"
Jul 23 17:50:49.878004 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:50:49.877973    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c993bf1b9acaba496a38c2b4ba451f2d4e1e62d8721c9d70cb0c4aed711ae240"
Jul 23 17:50:49.879083 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:50:49.879059    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7328-140135561016928" event={"ID":"795a84aa-326a-4290-961e-9af262c37f43","Type":"ContainerStarted","Data":"cf3fbe64fd9e27494baa2ef10745d2a3734cf716f516304369ce6e7148f5ee39"}
Jul 23 17:50:49.889906 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:50:49.889879    2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7328-140135561176112\" is forbidden: User \"system:node:ip-10-0-129-216.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-216.ec2.internal' and this object" podUID="74bea808-c60b-4b6c-aa38-31420cf780f1" pod="opendatahub/test-curl-7328-140135561176112"
Jul 23 17:50:50.766807 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:50:50.766769    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="74bea808-c60b-4b6c-aa38-31420cf780f1" path="/var/lib/kubelet/pods/74bea808-c60b-4b6c-aa38-31420cf780f1/volumes"
Jul 23 17:50:50.886266 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:50:50.886210    2577 generic.go:356] "Generic (PLEG): container finished" podID="795a84aa-326a-4290-961e-9af262c37f43" containerID="657287b506a9306525fb02c052741a26861111691f90af5635d8a2cdff4683a4" exitCode=0
Jul 23 17:50:50.886422 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:50:50.886301    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7328-140135561016928" event={"ID":"795a84aa-326a-4290-961e-9af262c37f43","Type":"ContainerDied","Data":"657287b506a9306525fb02c052741a26861111691f90af5635d8a2cdff4683a4"}
Jul 23 17:50:52.032746 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:50:52.032716    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7328-140135561016928"
Jul 23 17:50:52.058973 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:50:52.057500    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7328-140135561016928_795a84aa-326a-4290-961e-9af262c37f43/test-curl-7328-140135561016928/0.log"
Jul 23 17:50:52.083895 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:50:52.083863    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7328-140135561016928"]
Jul 23 17:50:52.087494 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:50:52.087468    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7328-140135561016928"]
Jul 23 17:50:52.124401 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:50:52.124378    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r7jp6\" (UniqueName: \"kubernetes.io/projected/795a84aa-326a-4290-961e-9af262c37f43-kube-api-access-r7jp6\") pod \"795a84aa-326a-4290-961e-9af262c37f43\" (UID: \"795a84aa-326a-4290-961e-9af262c37f43\") "
Jul 23 17:50:52.126508 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:50:52.126481    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/795a84aa-326a-4290-961e-9af262c37f43-kube-api-access-r7jp6" (OuterVolumeSpecName: "kube-api-access-r7jp6") pod "795a84aa-326a-4290-961e-9af262c37f43" (UID: "795a84aa-326a-4290-961e-9af262c37f43"). InnerVolumeSpecName "kube-api-access-r7jp6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 17:50:52.215179 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:50:52.215140    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7328-140135561016928"]
Jul 23 17:50:52.215678 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:50:52.215659    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="795a84aa-326a-4290-961e-9af262c37f43" containerName="test-curl-7328-140135561016928"
Jul 23 17:50:52.215773 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:50:52.215680    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="795a84aa-326a-4290-961e-9af262c37f43" containerName="test-curl-7328-140135561016928"
Jul 23 17:50:52.215829 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:50:52.215795    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="795a84aa-326a-4290-961e-9af262c37f43" containerName="test-curl-7328-140135561016928"
Jul 23 17:50:52.219014 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:50:52.218994    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7328-140135561016928"
Jul 23 17:50:52.225884 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:50:52.225857    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7328-140135561016928"]
Jul 23 17:50:52.226136 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:50:52.226113    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r7jp6\" (UniqueName: \"kubernetes.io/projected/795a84aa-326a-4290-961e-9af262c37f43-kube-api-access-r7jp6\") on node \"ip-10-0-129-216.ec2.internal\" DevicePath \"\""
Jul 23 17:50:52.327416 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:50:52.327383    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvmzk\" (UniqueName: \"kubernetes.io/projected/6387273c-e42f-4014-9011-cfb84329afa9-kube-api-access-gvmzk\") pod \"test-curl-7328-140135561016928\" (UID: \"6387273c-e42f-4014-9011-cfb84329afa9\") " pod="opendatahub/test-curl-7328-140135561016928"
Jul 23 17:50:52.428849 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:50:52.428816    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvmzk\" (UniqueName: \"kubernetes.io/projected/6387273c-e42f-4014-9011-cfb84329afa9-kube-api-access-gvmzk\") pod \"test-curl-7328-140135561016928\" (UID: \"6387273c-e42f-4014-9011-cfb84329afa9\") " pod="opendatahub/test-curl-7328-140135561016928"
Jul 23 17:50:52.437330 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:50:52.437299    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvmzk\" (UniqueName: \"kubernetes.io/projected/6387273c-e42f-4014-9011-cfb84329afa9-kube-api-access-gvmzk\") pod \"test-curl-7328-140135561016928\" (UID: \"6387273c-e42f-4014-9011-cfb84329afa9\") " pod="opendatahub/test-curl-7328-140135561016928"
Jul 23 17:50:52.517006 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:50:52.516920    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:50:52.529753 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:50:52.529730    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7328-140135561016928"
Jul 23 17:50:52.667111 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:50:52.667085    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7328-140135561016928"]
Jul 23 17:50:52.667805 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:50:52.667784    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6387273c_e42f_4014_9011_cfb84329afa9.slice/crio-3f001e192e25b32dcb1eaa376b609aecf14a2e6973d691db8c46c73430dc4c51 WatchSource:0}: Error finding container 3f001e192e25b32dcb1eaa376b609aecf14a2e6973d691db8c46c73430dc4c51: Status 404 returned error can't find the container with id 3f001e192e25b32dcb1eaa376b609aecf14a2e6973d691db8c46c73430dc4c51
Jul 23 17:50:52.715443 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:50:52.715418    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:50:52.767458 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:50:52.767396    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="795a84aa-326a-4290-961e-9af262c37f43" path="/var/lib/kubelet/pods/795a84aa-326a-4290-961e-9af262c37f43/volumes"
Jul 23 17:50:52.784730 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:50:52.784707    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:50:52.901035 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:50:52.901009    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7328-140135561016928"
Jul 23 17:50:52.901035 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:50:52.901033    2577 scope.go:117] "RemoveContainer" containerID="657287b506a9306525fb02c052741a26861111691f90af5635d8a2cdff4683a4"
Jul 23 17:50:52.902446 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:50:52.902417    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7328-140135561016928" event={"ID":"6387273c-e42f-4014-9011-cfb84329afa9","Type":"ContainerStarted","Data":"3f001e192e25b32dcb1eaa376b609aecf14a2e6973d691db8c46c73430dc4c51"}
Jul 23 17:50:53.912036 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:50:53.912003    2577 generic.go:356] "Generic (PLEG): container finished" podID="6387273c-e42f-4014-9011-cfb84329afa9" containerID="37590cca64a4b22dee91a366f542f582bd6e942a2343e03e2350e18d2d14cbfc" exitCode=0
Jul 23 17:50:53.912480 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:50:53.912083    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7328-140135561016928" event={"ID":"6387273c-e42f-4014-9011-cfb84329afa9","Type":"ContainerDied","Data":"37590cca64a4b22dee91a366f542f582bd6e942a2343e03e2350e18d2d14cbfc"}
Jul 23 17:50:55.069644 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:50:55.069618    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7328-140135561016928"
Jul 23 17:50:55.092904 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:50:55.092882    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7328-140135561016928_6387273c-e42f-4014-9011-cfb84329afa9/test-curl-7328-140135561016928/0.log"
Jul 23 17:50:55.125735 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:50:55.125705    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7328-140135561016928"]
Jul 23 17:50:55.130051 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:50:55.130029    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7328-140135561016928"]
Jul 23 17:50:55.153602 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:50:55.153570    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gvmzk\" (UniqueName: \"kubernetes.io/projected/6387273c-e42f-4014-9011-cfb84329afa9-kube-api-access-gvmzk\") pod \"6387273c-e42f-4014-9011-cfb84329afa9\" (UID: \"6387273c-e42f-4014-9011-cfb84329afa9\") "
Jul 23 17:50:55.155575 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:50:55.155553    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6387273c-e42f-4014-9011-cfb84329afa9-kube-api-access-gvmzk" (OuterVolumeSpecName: "kube-api-access-gvmzk") pod "6387273c-e42f-4014-9011-cfb84329afa9" (UID: "6387273c-e42f-4014-9011-cfb84329afa9"). InnerVolumeSpecName "kube-api-access-gvmzk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 17:50:55.259212 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:50:55.259176    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gvmzk\" (UniqueName: \"kubernetes.io/projected/6387273c-e42f-4014-9011-cfb84329afa9-kube-api-access-gvmzk\") on node \"ip-10-0-129-216.ec2.internal\" DevicePath \"\""
Jul 23 17:50:55.274703 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:50:55.274663    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7328-140135561017936"]
Jul 23 17:50:55.275148 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:50:55.275133    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6387273c-e42f-4014-9011-cfb84329afa9" containerName="test-curl-7328-140135561016928"
Jul 23 17:50:55.275206 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:50:55.275150    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="6387273c-e42f-4014-9011-cfb84329afa9" containerName="test-curl-7328-140135561016928"
Jul 23 17:50:55.275249 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:50:55.275225    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="6387273c-e42f-4014-9011-cfb84329afa9" containerName="test-curl-7328-140135561016928"
Jul 23 17:50:55.278105 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:50:55.278089    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7328-140135561017936"
Jul 23 17:50:55.291952 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:50:55.291925    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7328-140135561017936"]
Jul 23 17:50:55.360319 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:50:55.360288    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrp5m\" (UniqueName: \"kubernetes.io/projected/840f9e9b-1532-40c4-bf12-a2a376ff7ec4-kube-api-access-rrp5m\") pod \"test-curl-7328-140135561017936\" (UID: \"840f9e9b-1532-40c4-bf12-a2a376ff7ec4\") " pod="opendatahub/test-curl-7328-140135561017936"
Jul 23 17:50:55.460873 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:50:55.460840    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrp5m\" (UniqueName: \"kubernetes.io/projected/840f9e9b-1532-40c4-bf12-a2a376ff7ec4-kube-api-access-rrp5m\") pod \"test-curl-7328-140135561017936\" (UID: \"840f9e9b-1532-40c4-bf12-a2a376ff7ec4\") " pod="opendatahub/test-curl-7328-140135561017936"
Jul 23 17:50:55.469898 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:50:55.469867    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrp5m\" (UniqueName: \"kubernetes.io/projected/840f9e9b-1532-40c4-bf12-a2a376ff7ec4-kube-api-access-rrp5m\") pod \"test-curl-7328-140135561017936\" (UID: \"840f9e9b-1532-40c4-bf12-a2a376ff7ec4\") " pod="opendatahub/test-curl-7328-140135561017936"
Jul 23 17:50:55.588227 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:50:55.588139    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7328-140135561017936"
Jul 23 17:50:55.723892 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:50:55.723864    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7328-140135561017936"]
Jul 23 17:50:55.724991 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:50:55.724968    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod840f9e9b_1532_40c4_bf12_a2a376ff7ec4.slice/crio-2e22cef4ca740869f8f64e3eeba68623ac75af08d83fdb0de0f42e8697ff8175 WatchSource:0}: Error finding container 2e22cef4ca740869f8f64e3eeba68623ac75af08d83fdb0de0f42e8697ff8175: Status 404 returned error can't find the container with id 2e22cef4ca740869f8f64e3eeba68623ac75af08d83fdb0de0f42e8697ff8175
Jul 23 17:50:55.926828 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:50:55.926727    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7328-140135561017936" event={"ID":"840f9e9b-1532-40c4-bf12-a2a376ff7ec4","Type":"ContainerStarted","Data":"2e22cef4ca740869f8f64e3eeba68623ac75af08d83fdb0de0f42e8697ff8175"}
Jul 23 17:50:55.928034 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:50:55.928005    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3f001e192e25b32dcb1eaa376b609aecf14a2e6973d691db8c46c73430dc4c51"
Jul 23 17:50:55.928034 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:50:55.928033    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7328-140135561016928"
Jul 23 17:50:55.941428 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:50:55.941401    2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7328-140135561016928\" is forbidden: User \"system:node:ip-10-0-129-216.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-216.ec2.internal' and this object" podUID="6387273c-e42f-4014-9011-cfb84329afa9" pod="opendatahub/test-curl-7328-140135561016928"
Jul 23 17:50:55.952369 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:50:55.952337    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-b9d590/rate-test-model-b9d590-kserve-679779687f-jccfk" podUID="229d3b28-4807-4e27-8a48-74b2864154fc" containerName="main" containerID="cri-o://ee692dd0fbf98fdb6c20dc151e0eef04cc900c7ea0fe4df521bbb8b93e42545d" gracePeriod=45
Jul 23 17:50:56.211427 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:50:56.211402    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-b9d590/rate-test-model-b9d590-kserve-679779687f-jccfk"
Jul 23 17:50:56.231843 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:50:56.231804    2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7328-140135561016928\" is forbidden: User \"system:node:ip-10-0-129-216.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-216.ec2.internal' and this object" podUID="6387273c-e42f-4014-9011-cfb84329afa9" pod="opendatahub/test-curl-7328-140135561016928"
Jul 23 17:50:56.369947 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:50:56.369912    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/229d3b28-4807-4e27-8a48-74b2864154fc-tmp-dir\") pod \"229d3b28-4807-4e27-8a48-74b2864154fc\" (UID: \"229d3b28-4807-4e27-8a48-74b2864154fc\") "
Jul 23 17:50:56.369947 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:50:56.369946    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/229d3b28-4807-4e27-8a48-74b2864154fc-model-cache\") pod \"229d3b28-4807-4e27-8a48-74b2864154fc\" (UID: \"229d3b28-4807-4e27-8a48-74b2864154fc\") "
Jul 23 17:50:56.370199 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:50:56.369978    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l8zb2\" (UniqueName: \"kubernetes.io/projected/229d3b28-4807-4e27-8a48-74b2864154fc-kube-api-access-l8zb2\") pod \"229d3b28-4807-4e27-8a48-74b2864154fc\" (UID: \"229d3b28-4807-4e27-8a48-74b2864154fc\") "
Jul 23 17:50:56.370199 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:50:56.370042    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/229d3b28-4807-4e27-8a48-74b2864154fc-dshm\") pod \"229d3b28-4807-4e27-8a48-74b2864154fc\" (UID: \"229d3b28-4807-4e27-8a48-74b2864154fc\") "
Jul 23 17:50:56.370199 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:50:56.370170    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/229d3b28-4807-4e27-8a48-74b2864154fc-home\") pod \"229d3b28-4807-4e27-8a48-74b2864154fc\" (UID: \"229d3b28-4807-4e27-8a48-74b2864154fc\") "
Jul 23 17:50:56.370199 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:50:56.370194    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/229d3b28-4807-4e27-8a48-74b2864154fc-tls-certs\") pod \"229d3b28-4807-4e27-8a48-74b2864154fc\" (UID: \"229d3b28-4807-4e27-8a48-74b2864154fc\") "
Jul 23 17:50:56.370459 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:50:56.370242    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/229d3b28-4807-4e27-8a48-74b2864154fc-model-cache" (OuterVolumeSpecName: "model-cache") pod "229d3b28-4807-4e27-8a48-74b2864154fc" (UID: "229d3b28-4807-4e27-8a48-74b2864154fc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 17:50:56.370531 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:50:56.370487    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/229d3b28-4807-4e27-8a48-74b2864154fc-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "229d3b28-4807-4e27-8a48-74b2864154fc" (UID: "229d3b28-4807-4e27-8a48-74b2864154fc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 17:50:56.370531 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:50:56.370506    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/229d3b28-4807-4e27-8a48-74b2864154fc-home" (OuterVolumeSpecName: "home") pod "229d3b28-4807-4e27-8a48-74b2864154fc" (UID: "229d3b28-4807-4e27-8a48-74b2864154fc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 17:50:56.370631 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:50:56.370546    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/229d3b28-4807-4e27-8a48-74b2864154fc-model-cache\") on node \"ip-10-0-129-216.ec2.internal\" DevicePath \"\""
Jul 23 17:50:56.372218 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:50:56.372194    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/229d3b28-4807-4e27-8a48-74b2864154fc-dshm" (OuterVolumeSpecName: "dshm") pod "229d3b28-4807-4e27-8a48-74b2864154fc" (UID: "229d3b28-4807-4e27-8a48-74b2864154fc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 23 17:50:56.372349 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:50:56.372330    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/229d3b28-4807-4e27-8a48-74b2864154fc-kube-api-access-l8zb2" (OuterVolumeSpecName: "kube-api-access-l8zb2") pod "229d3b28-4807-4e27-8a48-74b2864154fc" (UID: "229d3b28-4807-4e27-8a48-74b2864154fc"). InnerVolumeSpecName "kube-api-access-l8zb2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 17:50:56.372414 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:50:56.372373    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/229d3b28-4807-4e27-8a48-74b2864154fc-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "229d3b28-4807-4e27-8a48-74b2864154fc" (UID: "229d3b28-4807-4e27-8a48-74b2864154fc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 17:50:56.471890 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:50:56.471817    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/229d3b28-4807-4e27-8a48-74b2864154fc-home\") on node \"ip-10-0-129-216.ec2.internal\" DevicePath \"\""
Jul 23 17:50:56.471890 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:50:56.471844    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/229d3b28-4807-4e27-8a48-74b2864154fc-tls-certs\") on node \"ip-10-0-129-216.ec2.internal\" DevicePath \"\""
Jul 23 17:50:56.471890 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:50:56.471854    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/229d3b28-4807-4e27-8a48-74b2864154fc-tmp-dir\") on node \"ip-10-0-129-216.ec2.internal\" DevicePath \"\""
Jul 23 17:50:56.471890 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:50:56.471862    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l8zb2\" (UniqueName: \"kubernetes.io/projected/229d3b28-4807-4e27-8a48-74b2864154fc-kube-api-access-l8zb2\") on node \"ip-10-0-129-216.ec2.internal\" DevicePath \"\""
Jul 23 17:50:56.471890 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:50:56.471871    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/229d3b28-4807-4e27-8a48-74b2864154fc-dshm\") on node \"ip-10-0-129-216.ec2.internal\" DevicePath \"\""
Jul 23 17:50:56.766869 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:50:56.766788    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6387273c-e42f-4014-9011-cfb84329afa9" path="/var/lib/kubelet/pods/6387273c-e42f-4014-9011-cfb84329afa9/volumes"
Jul 23 17:50:56.936390 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:50:56.936353    2577 generic.go:356] "Generic (PLEG): container finished" podID="229d3b28-4807-4e27-8a48-74b2864154fc" containerID="ee692dd0fbf98fdb6c20dc151e0eef04cc900c7ea0fe4df521bbb8b93e42545d" exitCode=0
Jul 23 17:50:56.936610 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:50:56.936405    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-b9d590/rate-test-model-b9d590-kserve-679779687f-jccfk"
Jul 23 17:50:56.936610 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:50:56.936447    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-b9d590/rate-test-model-b9d590-kserve-679779687f-jccfk" event={"ID":"229d3b28-4807-4e27-8a48-74b2864154fc","Type":"ContainerDied","Data":"ee692dd0fbf98fdb6c20dc151e0eef04cc900c7ea0fe4df521bbb8b93e42545d"}
Jul 23 17:50:56.936610 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:50:56.936485    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-b9d590/rate-test-model-b9d590-kserve-679779687f-jccfk" event={"ID":"229d3b28-4807-4e27-8a48-74b2864154fc","Type":"ContainerDied","Data":"463379c2f9616d93322f240a951276c511015edfefd42082b5675661506bef60"}
Jul 23 17:50:56.936610 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:50:56.936501    2577 scope.go:117] "RemoveContainer" containerID="ee692dd0fbf98fdb6c20dc151e0eef04cc900c7ea0fe4df521bbb8b93e42545d"
Jul 23 17:50:56.938461 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:50:56.938433    2577 generic.go:356] "Generic (PLEG): container finished" podID="840f9e9b-1532-40c4-bf12-a2a376ff7ec4" containerID="63e96f79eab478188eea31b6e4603d169eb4589492e4fce982709f9d4c0aa0d0" exitCode=0
Jul 23 17:50:56.938691 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:50:56.938634    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7328-140135561017936" event={"ID":"840f9e9b-1532-40c4-bf12-a2a376ff7ec4","Type":"ContainerDied","Data":"63e96f79eab478188eea31b6e4603d169eb4589492e4fce982709f9d4c0aa0d0"}
Jul 23 17:50:56.946720 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:50:56.946702    2577 scope.go:117] "RemoveContainer" containerID="ee692dd0fbf98fdb6c20dc151e0eef04cc900c7ea0fe4df521bbb8b93e42545d"
Jul 23 17:50:56.946993 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:50:56.946972    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee692dd0fbf98fdb6c20dc151e0eef04cc900c7ea0fe4df521bbb8b93e42545d\": container with ID starting with ee692dd0fbf98fdb6c20dc151e0eef04cc900c7ea0fe4df521bbb8b93e42545d not found: ID does not exist" containerID="ee692dd0fbf98fdb6c20dc151e0eef04cc900c7ea0fe4df521bbb8b93e42545d"
Jul 23 17:50:56.947069 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:50:56.947004    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee692dd0fbf98fdb6c20dc151e0eef04cc900c7ea0fe4df521bbb8b93e42545d"} err="failed to get container status \"ee692dd0fbf98fdb6c20dc151e0eef04cc900c7ea0fe4df521bbb8b93e42545d\": rpc error: code = NotFound desc = could not find container \"ee692dd0fbf98fdb6c20dc151e0eef04cc900c7ea0fe4df521bbb8b93e42545d\": container with ID starting with ee692dd0fbf98fdb6c20dc151e0eef04cc900c7ea0fe4df521bbb8b93e42545d not found: ID does not exist"
Jul 23 17:50:56.967554 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:50:56.967526    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-b9d590/rate-test-model-b9d590-kserve-679779687f-jccfk"]
Jul 23 17:50:56.973731 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:50:56.973704    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-b9d590/rate-test-model-b9d590-kserve-679779687f-jccfk"]
Jul 23 17:50:58.078502 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:50:58.078474    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7328-140135561017936"
Jul 23 17:50:58.099294 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:50:58.099265    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7328-140135561017936_840f9e9b-1532-40c4-bf12-a2a376ff7ec4/test-curl-7328-140135561017936/0.log"
Jul 23 17:50:58.125694 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:50:58.125667    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7328-140135561017936"]
Jul 23 17:50:58.128802 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:50:58.128780    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7328-140135561017936"]
Jul 23 17:50:58.185711 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:50:58.185689    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rrp5m\" (UniqueName: \"kubernetes.io/projected/840f9e9b-1532-40c4-bf12-a2a376ff7ec4-kube-api-access-rrp5m\") pod \"840f9e9b-1532-40c4-bf12-a2a376ff7ec4\" (UID: \"840f9e9b-1532-40c4-bf12-a2a376ff7ec4\") "
Jul 23 17:50:58.187692 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:50:58.187670    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/840f9e9b-1532-40c4-bf12-a2a376ff7ec4-kube-api-access-rrp5m" (OuterVolumeSpecName: "kube-api-access-rrp5m") pod "840f9e9b-1532-40c4-bf12-a2a376ff7ec4" (UID: "840f9e9b-1532-40c4-bf12-a2a376ff7ec4"). InnerVolumeSpecName "kube-api-access-rrp5m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 17:50:58.247426 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:50:58.247366    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7328-140135561017936"]
Jul 23 17:50:58.247771 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:50:58.247759    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="840f9e9b-1532-40c4-bf12-a2a376ff7ec4" containerName="test-curl-7328-140135561017936"
Jul 23 17:50:58.247824 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:50:58.247772    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="840f9e9b-1532-40c4-bf12-a2a376ff7ec4" containerName="test-curl-7328-140135561017936"
Jul 23 17:50:58.247824 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:50:58.247798    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="229d3b28-4807-4e27-8a48-74b2864154fc" containerName="main"
Jul 23 17:50:58.247824 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:50:58.247804    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="229d3b28-4807-4e27-8a48-74b2864154fc" containerName="main"
Jul 23 17:50:58.247933 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:50:58.247870    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="229d3b28-4807-4e27-8a48-74b2864154fc" containerName="main"
Jul 23 17:50:58.247933 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:50:58.247876    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="840f9e9b-1532-40c4-bf12-a2a376ff7ec4" containerName="test-curl-7328-140135561017936"
Jul 23 17:50:58.251986 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:50:58.251971    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7328-140135561017936"
Jul 23 17:50:58.255866 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:50:58.255840    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7328-140135561017936"]
Jul 23 17:50:58.286511 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:50:58.286481    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rrp5m\" (UniqueName: \"kubernetes.io/projected/840f9e9b-1532-40c4-bf12-a2a376ff7ec4-kube-api-access-rrp5m\") on node \"ip-10-0-129-216.ec2.internal\" DevicePath \"\""
Jul 23 17:50:58.387634 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:50:58.387605    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6cb5g\" (UniqueName: \"kubernetes.io/projected/6a591867-ffa4-44cc-9c42-93bda3a943ff-kube-api-access-6cb5g\") pod \"test-curl-7328-140135561017936\" (UID: \"6a591867-ffa4-44cc-9c42-93bda3a943ff\") " pod="opendatahub/test-curl-7328-140135561017936"
Jul 23 17:50:58.488927 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:50:58.488898    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6cb5g\" (UniqueName: \"kubernetes.io/projected/6a591867-ffa4-44cc-9c42-93bda3a943ff-kube-api-access-6cb5g\") pod \"test-curl-7328-140135561017936\" (UID: \"6a591867-ffa4-44cc-9c42-93bda3a943ff\") " pod="opendatahub/test-curl-7328-140135561017936"
Jul 23 17:50:58.497422 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:50:58.497399    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6cb5g\" (UniqueName: \"kubernetes.io/projected/6a591867-ffa4-44cc-9c42-93bda3a943ff-kube-api-access-6cb5g\") pod \"test-curl-7328-140135561017936\" (UID: \"6a591867-ffa4-44cc-9c42-93bda3a943ff\") " pod="opendatahub/test-curl-7328-140135561017936"
Jul 23 17:50:58.562213 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:50:58.562146    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7328-140135561017936"
Jul 23 17:50:58.768456 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:50:58.768417    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="229d3b28-4807-4e27-8a48-74b2864154fc" path="/var/lib/kubelet/pods/229d3b28-4807-4e27-8a48-74b2864154fc/volumes"
Jul 23 17:50:58.768992 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:50:58.768967    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="840f9e9b-1532-40c4-bf12-a2a376ff7ec4" path="/var/lib/kubelet/pods/840f9e9b-1532-40c4-bf12-a2a376ff7ec4/volumes"
Jul 23 17:50:58.899108 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:50:58.899085    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7328-140135561017936"]
Jul 23 17:50:58.901664 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:50:58.901634    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6a591867_ffa4_44cc_9c42_93bda3a943ff.slice/crio-30be6cef6315c0ecead47c12a004d71f9a6fda4b778a1ab8cc8a46338dcbbe0b WatchSource:0}: Error finding container 30be6cef6315c0ecead47c12a004d71f9a6fda4b778a1ab8cc8a46338dcbbe0b: Status 404 returned error can't find the container with id 30be6cef6315c0ecead47c12a004d71f9a6fda4b778a1ab8cc8a46338dcbbe0b
Jul 23 17:50:58.952967 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:50:58.952948    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7328-140135561017936"
Jul 23 17:50:58.953097 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:50:58.952968    2577 scope.go:117] "RemoveContainer" containerID="63e96f79eab478188eea31b6e4603d169eb4589492e4fce982709f9d4c0aa0d0"
Jul 23 17:50:58.954407 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:50:58.954372    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7328-140135561017936" event={"ID":"6a591867-ffa4-44cc-9c42-93bda3a943ff","Type":"ContainerStarted","Data":"30be6cef6315c0ecead47c12a004d71f9a6fda4b778a1ab8cc8a46338dcbbe0b"}
Jul 23 17:50:59.962264 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:50:59.962227    2577 generic.go:356] "Generic (PLEG): container finished" podID="6a591867-ffa4-44cc-9c42-93bda3a943ff" containerID="1aaf66c2c45fd35ac9ed91fd64eaf6755a0b03eed8ee75732b8c215687f1eade" exitCode=0
Jul 23 17:50:59.962639 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:50:59.962321    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7328-140135561017936" event={"ID":"6a591867-ffa4-44cc-9c42-93bda3a943ff","Type":"ContainerDied","Data":"1aaf66c2c45fd35ac9ed91fd64eaf6755a0b03eed8ee75732b8c215687f1eade"}
Jul 23 17:51:01.109956 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:51:01.109932    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7328-140135561017936"
Jul 23 17:51:01.136499 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:51:01.136468    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7328-140135561017936_6a591867-ffa4-44cc-9c42-93bda3a943ff/test-curl-7328-140135561017936/0.log"
Jul 23 17:51:01.164740 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:51:01.164712    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7328-140135561017936"]
Jul 23 17:51:01.167136 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:51:01.167114    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7328-140135561017936"]
Jul 23 17:51:01.222202 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:51:01.222138    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6cb5g\" (UniqueName: \"kubernetes.io/projected/6a591867-ffa4-44cc-9c42-93bda3a943ff-kube-api-access-6cb5g\") pod \"6a591867-ffa4-44cc-9c42-93bda3a943ff\" (UID: \"6a591867-ffa4-44cc-9c42-93bda3a943ff\") "
Jul 23 17:51:01.224133 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:51:01.224111    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a591867-ffa4-44cc-9c42-93bda3a943ff-kube-api-access-6cb5g" (OuterVolumeSpecName: "kube-api-access-6cb5g") pod "6a591867-ffa4-44cc-9c42-93bda3a943ff" (UID: "6a591867-ffa4-44cc-9c42-93bda3a943ff"). InnerVolumeSpecName "kube-api-access-6cb5g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 17:51:01.323673 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:51:01.323636    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6cb5g\" (UniqueName: \"kubernetes.io/projected/6a591867-ffa4-44cc-9c42-93bda3a943ff-kube-api-access-6cb5g\") on node \"ip-10-0-129-216.ec2.internal\" DevicePath \"\""
Jul 23 17:51:01.365832 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:51:01.365792    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-b9d590-5lpnj"]
Jul 23 17:51:01.366435 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:51:01.366414    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a591867-ffa4-44cc-9c42-93bda3a943ff" containerName="test-curl-7328-140135561017936"
Jul 23 17:51:01.366499 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:51:01.366436    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a591867-ffa4-44cc-9c42-93bda3a943ff" containerName="test-curl-7328-140135561017936"
Jul 23 17:51:01.366545 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:51:01.366506    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="6a591867-ffa4-44cc-9c42-93bda3a943ff" containerName="test-curl-7328-140135561017936"
Jul 23 17:51:01.368733 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:51:01.368715    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-b9d590-5lpnj"
Jul 23 17:51:01.371460 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:51:01.371437    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7wfmw\""
Jul 23 17:51:01.378359 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:51:01.378330    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-b9d590-5lpnj"]
Jul 23 17:51:01.424042 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:51:01.424011    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f842a7f2-a081-42a6-b891-462788fc2f18-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-b9d590-5lpnj\" (UID: \"f842a7f2-a081-42a6-b891-462788fc2f18\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-b9d590-5lpnj"
Jul 23 17:51:01.524466 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:51:01.524372    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f842a7f2-a081-42a6-b891-462788fc2f18-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-b9d590-5lpnj\" (UID: \"f842a7f2-a081-42a6-b891-462788fc2f18\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-b9d590-5lpnj"
Jul 23 17:51:01.524983 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:51:01.524961    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f842a7f2-a081-42a6-b891-462788fc2f18-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-b9d590-5lpnj\" (UID: \"f842a7f2-a081-42a6-b891-462788fc2f18\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-b9d590-5lpnj"
Jul 23 17:51:01.680839 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:51:01.680797    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-b9d590-5lpnj"
Jul 23 17:51:01.823714 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:51:01.823682    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-b9d590-5lpnj"]
Jul 23 17:51:01.824759 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:51:01.824728    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf842a7f2_a081_42a6_b891_462788fc2f18.slice/crio-793d3ed0e69443dd367849a0f0087d42fbc42d2384c69a6ec0dba109d2971aaa WatchSource:0}: Error finding container 793d3ed0e69443dd367849a0f0087d42fbc42d2384c69a6ec0dba109d2971aaa: Status 404 returned error can't find the container with id 793d3ed0e69443dd367849a0f0087d42fbc42d2384c69a6ec0dba109d2971aaa
Jul 23 17:51:01.968827 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:51:01.968793    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:51:01.975443 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:51:01.975410    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-b9d590-5lpnj" event={"ID":"f842a7f2-a081-42a6-b891-462788fc2f18","Type":"ContainerStarted","Data":"793d3ed0e69443dd367849a0f0087d42fbc42d2384c69a6ec0dba109d2971aaa"}
Jul 23 17:51:01.977000 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:51:01.976982    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="30be6cef6315c0ecead47c12a004d71f9a6fda4b778a1ab8cc8a46338dcbbe0b"
Jul 23 17:51:01.977000 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:51:01.977003    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7328-140135561017936"
Jul 23 17:51:01.986956 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:51:01.986930    2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7328-140135561017936\" is forbidden: User \"system:node:ip-10-0-129-216.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-216.ec2.internal' and this object" podUID="6a591867-ffa4-44cc-9c42-93bda3a943ff" pod="opendatahub/test-curl-7328-140135561017936"
Jul 23 17:51:02.070687 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:51:02.070653    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:51:02.767234 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:51:02.767203    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6a591867-ffa4-44cc-9c42-93bda3a943ff" path="/var/lib/kubelet/pods/6a591867-ffa4-44cc-9c42-93bda3a943ff/volumes"
Jul 23 17:51:02.990759 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:51:02.990723    2577 generic.go:356] "Generic (PLEG): container finished" podID="f842a7f2-a081-42a6-b891-462788fc2f18" containerID="6a05363204801e7cc11cfd01b3320a5df0c3100892eeb352648d7814cd45fd00" exitCode=0
Jul 23 17:51:02.990974 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:51:02.990802    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-b9d590-5lpnj" event={"ID":"f842a7f2-a081-42a6-b891-462788fc2f18","Type":"ContainerDied","Data":"6a05363204801e7cc11cfd01b3320a5df0c3100892eeb352648d7814cd45fd00"}
Jul 23 17:51:04.134152 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:51:04.134127    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-b9d590-5lpnj"
Jul 23 17:51:04.142559 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:51:04.142536    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f842a7f2-a081-42a6-b891-462788fc2f18-maas-api-service-ca\") pod \"f842a7f2-a081-42a6-b891-462788fc2f18\" (UID: \"f842a7f2-a081-42a6-b891-462788fc2f18\") "
Jul 23 17:51:04.142827 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:51:04.142807    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f842a7f2-a081-42a6-b891-462788fc2f18-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f842a7f2-a081-42a6-b891-462788fc2f18" (UID: "f842a7f2-a081-42a6-b891-462788fc2f18"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 17:51:04.243477 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:51:04.243446    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f842a7f2-a081-42a6-b891-462788fc2f18-maas-api-service-ca\") on node \"ip-10-0-129-216.ec2.internal\" DevicePath \"\""
Jul 23 17:51:05.004394 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:51:05.004356    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-b9d590-5lpnj" event={"ID":"f842a7f2-a081-42a6-b891-462788fc2f18","Type":"ContainerDied","Data":"793d3ed0e69443dd367849a0f0087d42fbc42d2384c69a6ec0dba109d2971aaa"}
Jul 23 17:51:05.004543 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:51:05.004406    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="793d3ed0e69443dd367849a0f0087d42fbc42d2384c69a6ec0dba109d2971aaa"
Jul 23 17:51:05.004543 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:51:05.004371    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-b9d590-5lpnj"
Jul 23 17:51:05.141408 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:51:05.141312    2577 scope.go:117] "RemoveContainer" containerID="a25d9be00a507c87ec52feff1be38ba8a665fa3bda88c34cb9743dc5691a49f0"
Jul 23 17:51:11.917943 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:51:11.917863    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-b9d590-ffdc5bcb6-qggkj"]
Jul 23 17:51:11.918440 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:51:11.918119    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-b9d590-ffdc5bcb6-qggkj" podUID="f00d0bc9-60c8-4b62-9622-15c089d173c9" containerName="maas-api" containerID="cri-o://7f6c84397209d9ce82322749fed3e58c32e38a46c71adb9365c5511c382bd30e" gracePeriod=30
Jul 23 17:51:11.976111 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:51:11.976077    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-b9d590-29747145-l4rqc"]
Jul 23 17:51:11.980502 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:51:11.980475    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-b9d590-29747145-l4rqc"]
Jul 23 17:51:11.991688 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:51:11.991665    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:51:12.045466 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:51:12.045432    2577 generic.go:356] "Generic (PLEG): container finished" podID="f00d0bc9-60c8-4b62-9622-15c089d173c9" containerID="7f6c84397209d9ce82322749fed3e58c32e38a46c71adb9365c5511c382bd30e" exitCode=0
Jul 23 17:51:12.045598 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:51:12.045500    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-b9d590-ffdc5bcb6-qggkj" event={"ID":"f00d0bc9-60c8-4b62-9622-15c089d173c9","Type":"ContainerDied","Data":"7f6c84397209d9ce82322749fed3e58c32e38a46c71adb9365c5511c382bd30e"}
Jul 23 17:51:12.174877 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:51:12.174819    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-b9d590-ffdc5bcb6-qggkj"
Jul 23 17:51:12.209376 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:51:12.209343    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f5z68\" (UniqueName: \"kubernetes.io/projected/f00d0bc9-60c8-4b62-9622-15c089d173c9-kube-api-access-f5z68\") pod \"f00d0bc9-60c8-4b62-9622-15c089d173c9\" (UID: \"f00d0bc9-60c8-4b62-9622-15c089d173c9\") "
Jul 23 17:51:12.209544 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:51:12.209411    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f00d0bc9-60c8-4b62-9622-15c089d173c9-openshift-service-ca\") pod \"f00d0bc9-60c8-4b62-9622-15c089d173c9\" (UID: \"f00d0bc9-60c8-4b62-9622-15c089d173c9\") "
Jul 23 17:51:12.209544 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:51:12.209458    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f00d0bc9-60c8-4b62-9622-15c089d173c9-maas-api-tls\") pod \"f00d0bc9-60c8-4b62-9622-15c089d173c9\" (UID: \"f00d0bc9-60c8-4b62-9622-15c089d173c9\") "
Jul 23 17:51:12.209790 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:51:12.209762    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f00d0bc9-60c8-4b62-9622-15c089d173c9-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "f00d0bc9-60c8-4b62-9622-15c089d173c9" (UID: "f00d0bc9-60c8-4b62-9622-15c089d173c9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 17:51:12.211563 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:51:12.211536    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f00d0bc9-60c8-4b62-9622-15c089d173c9-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "f00d0bc9-60c8-4b62-9622-15c089d173c9" (UID: "f00d0bc9-60c8-4b62-9622-15c089d173c9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 17:51:12.211671 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:51:12.211577    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f00d0bc9-60c8-4b62-9622-15c089d173c9-kube-api-access-f5z68" (OuterVolumeSpecName: "kube-api-access-f5z68") pod "f00d0bc9-60c8-4b62-9622-15c089d173c9" (UID: "f00d0bc9-60c8-4b62-9622-15c089d173c9"). InnerVolumeSpecName "kube-api-access-f5z68". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 17:51:12.311090 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:51:12.311060    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f5z68\" (UniqueName: \"kubernetes.io/projected/f00d0bc9-60c8-4b62-9622-15c089d173c9-kube-api-access-f5z68\") on node \"ip-10-0-129-216.ec2.internal\" DevicePath \"\""
Jul 23 17:51:12.311090 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:51:12.311088    2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f00d0bc9-60c8-4b62-9622-15c089d173c9-openshift-service-ca\") on node \"ip-10-0-129-216.ec2.internal\" DevicePath \"\""
Jul 23 17:51:12.311090 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:51:12.311098    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f00d0bc9-60c8-4b62-9622-15c089d173c9-maas-api-tls\") on node \"ip-10-0-129-216.ec2.internal\" DevicePath \"\""
Jul 23 17:51:12.767793 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:51:12.767751    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="41038e83-4fdc-4c45-a443-b0778b85e7b0" path="/var/lib/kubelet/pods/41038e83-4fdc-4c45-a443-b0778b85e7b0/volumes"
Jul 23 17:51:13.053443 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:51:13.053364    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-b9d590-ffdc5bcb6-qggkj"
Jul 23 17:51:13.053443 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:51:13.053413    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-b9d590-ffdc5bcb6-qggkj" event={"ID":"f00d0bc9-60c8-4b62-9622-15c089d173c9","Type":"ContainerDied","Data":"9f7be1fc080c7c89faf3ed7d447eef9aa1be307cbe7d06ea0ec3c1b0c2fbad95"}
Jul 23 17:51:13.053890 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:51:13.053474    2577 scope.go:117] "RemoveContainer" containerID="7f6c84397209d9ce82322749fed3e58c32e38a46c71adb9365c5511c382bd30e"
Jul 23 17:51:13.074736 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:51:13.074710    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-b9d590-ffdc5bcb6-qggkj"]
Jul 23 17:51:13.080426 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:51:13.080405    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-b9d590-ffdc5bcb6-qggkj"]
Jul 23 17:51:14.767825 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:51:14.767790    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f00d0bc9-60c8-4b62-9622-15c089d173c9" path="/var/lib/kubelet/pods/f00d0bc9-60c8-4b62-9622-15c089d173c9/volumes"
Jul 23 17:51:16.463545 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:51:16.463505    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-b9d590-5lpnj"]
Jul 23 17:51:16.466729 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:51:16.466700    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-b9d590-5lpnj"]
Jul 23 17:51:16.482226 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:51:16.482194    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:51:16.766191 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:51:16.766115    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f842a7f2-a081-42a6-b891-462788fc2f18" path="/var/lib/kubelet/pods/f842a7f2-a081-42a6-b891-462788fc2f18/volumes"
Jul 23 17:51:16.921355 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:51:16.921318    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:51:23.372652 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:51:23.372613    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:51:23.460725 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:51:23.460685    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:51:23.663497 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:51:23.663421    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:51:23.932457 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:51:23.932382    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:51:24.007912 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:51:24.007874    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-9c4020-69p27"]
Jul 23 17:51:24.008338 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:51:24.008324    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f842a7f2-a081-42a6-b891-462788fc2f18" containerName="revoke-keys"
Jul 23 17:51:24.008401 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:51:24.008340    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="f842a7f2-a081-42a6-b891-462788fc2f18" containerName="revoke-keys"
Jul 23 17:51:24.008401 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:51:24.008351    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f00d0bc9-60c8-4b62-9622-15c089d173c9" containerName="maas-api"
Jul 23 17:51:24.008401 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:51:24.008357    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="f00d0bc9-60c8-4b62-9622-15c089d173c9" containerName="maas-api"
Jul 23 17:51:24.008514 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:51:24.008412    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="f842a7f2-a081-42a6-b891-462788fc2f18" containerName="revoke-keys"
Jul 23 17:51:24.008514 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:51:24.008423    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="f00d0bc9-60c8-4b62-9622-15c089d173c9" containerName="maas-api"
Jul 23 17:51:24.011706 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:51:24.011689    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-9c4020-69p27"
Jul 23 17:51:24.014112 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:51:24.014088    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7wfmw\""
Jul 23 17:51:24.019098 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:51:24.019073    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-9c4020-69p27"]
Jul 23 17:51:24.115555 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:51:24.115522    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae10da88-21e3-4d9b-a7ea-f888503dafac-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-9c4020-69p27\" (UID: \"ae10da88-21e3-4d9b-a7ea-f888503dafac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-9c4020-69p27"
Jul 23 17:51:24.216379 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:51:24.216349    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae10da88-21e3-4d9b-a7ea-f888503dafac-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-9c4020-69p27\" (UID: \"ae10da88-21e3-4d9b-a7ea-f888503dafac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-9c4020-69p27"
Jul 23 17:51:24.217031 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:51:24.217004    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae10da88-21e3-4d9b-a7ea-f888503dafac-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-9c4020-69p27\" (UID: \"ae10da88-21e3-4d9b-a7ea-f888503dafac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-9c4020-69p27"
Jul 23 17:51:24.324635 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:51:24.324600    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-9c4020-69p27"
Jul 23 17:51:24.583623 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:51:24.583507    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:51:24.667573 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:51:24.667546    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-9c4020-69p27"]
Jul 23 17:51:24.669133 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:51:24.669109    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae10da88_21e3_4d9b_a7ea_f888503dafac.slice/crio-53be7cfcb4dd8d0b43351164bb208514763bb553ce16fee7aacd23dc87dcbfd1 WatchSource:0}: Error finding container 53be7cfcb4dd8d0b43351164bb208514763bb553ce16fee7aacd23dc87dcbfd1: Status 404 returned error can't find the container with id 53be7cfcb4dd8d0b43351164bb208514763bb553ce16fee7aacd23dc87dcbfd1
Jul 23 17:51:24.689314 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:51:24.689292    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:51:25.123066 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:51:25.122967    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-9c4020-69p27" event={"ID":"ae10da88-21e3-4d9b-a7ea-f888503dafac","Type":"ContainerStarted","Data":"2a1f04009585a754263814b631c90eea388baf2a66039db54fe81def535f446d"}
Jul 23 17:51:25.123066 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:51:25.123026    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-9c4020-69p27" event={"ID":"ae10da88-21e3-4d9b-a7ea-f888503dafac","Type":"ContainerStarted","Data":"53be7cfcb4dd8d0b43351164bb208514763bb553ce16fee7aacd23dc87dcbfd1"}
Jul 23 17:51:25.143562 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:51:25.143511    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-9c4020-69p27" podStartSLOduration=2.143495905 podStartE2EDuration="2.143495905s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 17:51:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 17:51:25.140614192 +0000 UTC m=+2780.910899710" watchObservedRunningTime="2026-07-23 17:51:25.143495905 +0000 UTC m=+2780.913781423"
Jul 23 17:51:26.131240 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:51:26.131212    2577 generic.go:356] "Generic (PLEG): container finished" podID="ae10da88-21e3-4d9b-a7ea-f888503dafac" containerID="2a1f04009585a754263814b631c90eea388baf2a66039db54fe81def535f446d" exitCode=0
Jul 23 17:51:26.131649 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:51:26.131292    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-9c4020-69p27" event={"ID":"ae10da88-21e3-4d9b-a7ea-f888503dafac","Type":"ContainerDied","Data":"2a1f04009585a754263814b631c90eea388baf2a66039db54fe81def535f446d"}
Jul 23 17:51:27.272609 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:51:27.272583    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-9c4020-69p27"
Jul 23 17:51:27.345007 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:51:27.344976    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae10da88-21e3-4d9b-a7ea-f888503dafac-maas-api-service-ca\") pod \"ae10da88-21e3-4d9b-a7ea-f888503dafac\" (UID: \"ae10da88-21e3-4d9b-a7ea-f888503dafac\") "
Jul 23 17:51:27.345395 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:51:27.345369    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae10da88-21e3-4d9b-a7ea-f888503dafac-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ae10da88-21e3-4d9b-a7ea-f888503dafac" (UID: "ae10da88-21e3-4d9b-a7ea-f888503dafac"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 17:51:27.446452 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:51:27.446425    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae10da88-21e3-4d9b-a7ea-f888503dafac-maas-api-service-ca\") on node \"ip-10-0-129-216.ec2.internal\" DevicePath \"\""
Jul 23 17:51:28.144236 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:51:28.144202    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-9c4020-69p27"
Jul 23 17:51:28.144425 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:51:28.144244    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-9c4020-69p27" event={"ID":"ae10da88-21e3-4d9b-a7ea-f888503dafac","Type":"ContainerDied","Data":"53be7cfcb4dd8d0b43351164bb208514763bb553ce16fee7aacd23dc87dcbfd1"}
Jul 23 17:51:28.144425 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:51:28.144306    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="53be7cfcb4dd8d0b43351164bb208514763bb553ce16fee7aacd23dc87dcbfd1"
Jul 23 17:51:34.617583 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:51:34.617549    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-9c4020-5869f85b5b-v52vg"]
Jul 23 17:51:34.618112 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:51:34.617757    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-9c4020-5869f85b5b-v52vg" podUID="b73f6c46-6811-48e3-9918-4b7c2b98595d" containerName="maas-api" containerID="cri-o://dbfd8e1e67487bdff231ce897e5c54c292f9c865385a096e93a9056d287c9936" gracePeriod=30
Jul 23 17:51:34.641991 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:51:34.641952    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:51:34.872511 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:51:34.872455    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-9c4020-5869f85b5b-v52vg"
Jul 23 17:51:35.013477 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:51:35.013448    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rfwj8\" (UniqueName: \"kubernetes.io/projected/b73f6c46-6811-48e3-9918-4b7c2b98595d-kube-api-access-rfwj8\") pod \"b73f6c46-6811-48e3-9918-4b7c2b98595d\" (UID: \"b73f6c46-6811-48e3-9918-4b7c2b98595d\") "
Jul 23 17:51:35.013691 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:51:35.013537    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b73f6c46-6811-48e3-9918-4b7c2b98595d-maas-api-tls\") pod \"b73f6c46-6811-48e3-9918-4b7c2b98595d\" (UID: \"b73f6c46-6811-48e3-9918-4b7c2b98595d\") "
Jul 23 17:51:35.013691 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:51:35.013584    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b73f6c46-6811-48e3-9918-4b7c2b98595d-openshift-service-ca\") pod \"b73f6c46-6811-48e3-9918-4b7c2b98595d\" (UID: \"b73f6c46-6811-48e3-9918-4b7c2b98595d\") "
Jul 23 17:51:35.013920 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:51:35.013895    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b73f6c46-6811-48e3-9918-4b7c2b98595d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "b73f6c46-6811-48e3-9918-4b7c2b98595d" (UID: "b73f6c46-6811-48e3-9918-4b7c2b98595d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 23 17:51:35.015705 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:51:35.015681    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b73f6c46-6811-48e3-9918-4b7c2b98595d-kube-api-access-rfwj8" (OuterVolumeSpecName: "kube-api-access-rfwj8") pod "b73f6c46-6811-48e3-9918-4b7c2b98595d" (UID: "b73f6c46-6811-48e3-9918-4b7c2b98595d"). InnerVolumeSpecName "kube-api-access-rfwj8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 23 17:51:35.015822 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:51:35.015708    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b73f6c46-6811-48e3-9918-4b7c2b98595d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "b73f6c46-6811-48e3-9918-4b7c2b98595d" (UID: "b73f6c46-6811-48e3-9918-4b7c2b98595d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 23 17:51:35.115134 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:51:35.115103    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b73f6c46-6811-48e3-9918-4b7c2b98595d-maas-api-tls\") on node \"ip-10-0-129-216.ec2.internal\" DevicePath \"\""
Jul 23 17:51:35.115134 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:51:35.115130    2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b73f6c46-6811-48e3-9918-4b7c2b98595d-openshift-service-ca\") on node \"ip-10-0-129-216.ec2.internal\" DevicePath \"\""
Jul 23 17:51:35.115134 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:51:35.115142    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rfwj8\" (UniqueName: \"kubernetes.io/projected/b73f6c46-6811-48e3-9918-4b7c2b98595d-kube-api-access-rfwj8\") on node \"ip-10-0-129-216.ec2.internal\" DevicePath \"\""
Jul 23 17:51:35.186899 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:51:35.186816    2577 generic.go:356] "Generic (PLEG): container finished" podID="b73f6c46-6811-48e3-9918-4b7c2b98595d" containerID="dbfd8e1e67487bdff231ce897e5c54c292f9c865385a096e93a9056d287c9936" exitCode=0
Jul 23 17:51:35.186899 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:51:35.186861    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-9c4020-5869f85b5b-v52vg"
Jul 23 17:51:35.187139 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:51:35.186910    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-9c4020-5869f85b5b-v52vg" event={"ID":"b73f6c46-6811-48e3-9918-4b7c2b98595d","Type":"ContainerDied","Data":"dbfd8e1e67487bdff231ce897e5c54c292f9c865385a096e93a9056d287c9936"}
Jul 23 17:51:35.187139 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:51:35.186968    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-9c4020-5869f85b5b-v52vg" event={"ID":"b73f6c46-6811-48e3-9918-4b7c2b98595d","Type":"ContainerDied","Data":"f9dc40c1122cd2bb76ef9b5a1d427ea21bc8faf99243eba989608bfc7f158251"}
Jul 23 17:51:35.187139 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:51:35.187000    2577 scope.go:117] "RemoveContainer" containerID="dbfd8e1e67487bdff231ce897e5c54c292f9c865385a096e93a9056d287c9936"
Jul 23 17:51:35.197548 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:51:35.197529    2577 scope.go:117] "RemoveContainer" containerID="dbfd8e1e67487bdff231ce897e5c54c292f9c865385a096e93a9056d287c9936"
Jul 23 17:51:35.197802 ip-10-0-129-216 kubenswrapper[2577]: E0723 17:51:35.197783    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dbfd8e1e67487bdff231ce897e5c54c292f9c865385a096e93a9056d287c9936\": container with ID starting with dbfd8e1e67487bdff231ce897e5c54c292f9c865385a096e93a9056d287c9936 not found: ID does not exist" containerID="dbfd8e1e67487bdff231ce897e5c54c292f9c865385a096e93a9056d287c9936"
Jul 23 17:51:35.197851 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:51:35.197810    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dbfd8e1e67487bdff231ce897e5c54c292f9c865385a096e93a9056d287c9936"} err="failed to get container status \"dbfd8e1e67487bdff231ce897e5c54c292f9c865385a096e93a9056d287c9936\": rpc error: code = NotFound desc = could not find container \"dbfd8e1e67487bdff231ce897e5c54c292f9c865385a096e93a9056d287c9936\": container with ID starting with dbfd8e1e67487bdff231ce897e5c54c292f9c865385a096e93a9056d287c9936 not found: ID does not exist"
Jul 23 17:51:35.213696 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:51:35.213671    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-9c4020-5869f85b5b-v52vg"]
Jul 23 17:51:35.217954 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:51:35.217934    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-9c4020-5869f85b5b-v52vg"]
Jul 23 17:51:36.766513 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:51:36.766481    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b73f6c46-6811-48e3-9918-4b7c2b98595d" path="/var/lib/kubelet/pods/b73f6c46-6811-48e3-9918-4b7c2b98595d/volumes"
Jul 23 17:51:39.106757 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:51:39.106720    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-9c4020-69p27"]
Jul 23 17:51:39.109680 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:51:39.109650    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-9c4020-69p27"]
Jul 23 17:51:40.081864 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:51:40.081831    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:51:40.184587 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:51:40.184556    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:51:40.766188 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:51:40.766154    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ae10da88-21e3-4d9b-a7ea-f888503dafac" path="/var/lib/kubelet/pods/ae10da88-21e3-4d9b-a7ea-f888503dafac/volumes"
Jul 23 17:51:45.847862 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:51:45.847829    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:51:45.950378 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:51:45.950342    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:51:46.132386 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:51:46.132304    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:51:46.434744 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:51:46.434648    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:51:52.526498 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:51:52.526469    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-6svpm_4e7eff36-6da5-43e5-be32-820f601c91c7/manager/0.log"
Jul 23 17:51:52.649494 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:51:52.649460    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7979b9b8c-vm92w_3f763d30-b0dc-4dd9-b0b1-b91ee5c631f9/manager/0.log"
Jul 23 17:51:52.912876 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:51:52.912805    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-z4mjc_aa92e815-3d31-4472-b7ef-ec1f76756bce/manager/0.log"
Jul 23 17:51:53.025786 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:51:53.025760    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-f9b469455-874ln_317747fa-9885-43a3-a464-bbdb37ac9a6c/copy-manifests/0.log"
Jul 23 17:51:53.038813 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:51:53.038780    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-f9b469455-874ln_317747fa-9885-43a3-a464-bbdb37ac9a6c/manager/0.log"
Jul 23 17:51:53.640142 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:51:53.640106    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-7fc9b4c8bc-ng7g6_62efdc6c-a1cf-458e-9e9d-190a8130972b/maas-api/0.log"
Jul 23 17:51:53.988056 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:51:53.988029    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29747145-64pnm_78cb17f5-738b-40f1-b40b-458ed15c8b06/cleanup/0.log"
Jul 23 17:51:54.106306 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:51:54.106277    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-llvh2_983722e8-323d-4894-9391-513f0692387e/postgres/0.log"
Jul 23 17:51:54.886539 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:51:54.886506    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759zx982_67c1a8e8-ea9e-4d80-b4e8-cecbb603a745/extract/0.log"
Jul 23 17:51:54.894069 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:51:54.894050    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759zx982_67c1a8e8-ea9e-4d80-b4e8-cecbb603a745/util/0.log"
Jul 23 17:51:54.901419 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:51:54.901398    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759zx982_67c1a8e8-ea9e-4d80-b4e8-cecbb603a745/pull/0.log"
Jul 23 17:51:55.012266 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:51:55.012229    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xpmjs_62a7ca3d-ef60-422c-bef3-627044060c5f/pull/0.log"
Jul 23 17:51:55.019516 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:51:55.019498    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xpmjs_62a7ca3d-ef60-422c-bef3-627044060c5f/extract/0.log"
Jul 23 17:51:55.026923 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:51:55.026900    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xpmjs_62a7ca3d-ef60-422c-bef3-627044060c5f/util/0.log"
Jul 23 17:51:55.141730 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:51:55.141660    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xchhq_283546e1-df7c-49a0-a72a-2318766fc257/util/0.log"
Jul 23 17:51:55.149618 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:51:55.149595    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xchhq_283546e1-df7c-49a0-a72a-2318766fc257/pull/0.log"
Jul 23 17:51:55.156892 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:51:55.156872    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xchhq_283546e1-df7c-49a0-a72a-2318766fc257/extract/0.log"
Jul 23 17:51:55.269588 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:51:55.269559    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jqkqj_f0f53fe8-060d-4ddc-94a6-c361cc5d8688/util/0.log"
Jul 23 17:51:55.276658 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:51:55.276636    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jqkqj_f0f53fe8-060d-4ddc-94a6-c361cc5d8688/pull/0.log"
Jul 23 17:51:55.283610 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:51:55.283591    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jqkqj_f0f53fe8-060d-4ddc-94a6-c361cc5d8688/extract/0.log"
Jul 23 17:51:55.677861 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:51:55.677831    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-qtrml_35d67fdc-c2d6-4774-9912-1e088f3487bd/manager/0.log"
Jul 23 17:51:55.915432 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:51:55.915402    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-bd4jn_d572c939-0f51-4f05-950c-fe8ce11eacd3/registry-server/0.log"
Jul 23 17:51:56.040885 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:51:56.040856    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-hxc6t_dbd9fff4-9aef-4496-b026-e7cc838cd76f/manager/0.log"
Jul 23 17:51:56.282952 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:51:56.282915    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-pbmw7_f702fd02-0777-4c66-acf3-2463cd91eafe/manager/0.log"
Jul 23 17:51:57.394959 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:51:57.394927    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5bf47cd988-tb75s_7f87b7f7-f222-43da-9f48-37f88ae96fc5/payload-processing/0.log"
Jul 23 17:51:58.115639 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:51:58.115609    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-5ffcv_76383eb1-5f14-4225-9233-deac178d77ba/main/0.log"
Jul 23 17:51:58.360710 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:51:58.360674    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-msdd5_02ac9ef4-9ce4-49b5-8a1b-4b337b4ed85e/main/0.log"
Jul 23 17:52:04.121112 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:52:04.121074    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-5mz8v/must-gather-dz64n"]
Jul 23 17:52:04.121586 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:52:04.121494    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae10da88-21e3-4d9b-a7ea-f888503dafac" containerName="revoke-keys"
Jul 23 17:52:04.121586 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:52:04.121506    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae10da88-21e3-4d9b-a7ea-f888503dafac" containerName="revoke-keys"
Jul 23 17:52:04.121586 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:52:04.121515    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b73f6c46-6811-48e3-9918-4b7c2b98595d" containerName="maas-api"
Jul 23 17:52:04.121586 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:52:04.121521    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="b73f6c46-6811-48e3-9918-4b7c2b98595d" containerName="maas-api"
Jul 23 17:52:04.121586 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:52:04.121585    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="b73f6c46-6811-48e3-9918-4b7c2b98595d" containerName="maas-api"
Jul 23 17:52:04.121753 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:52:04.121596    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="ae10da88-21e3-4d9b-a7ea-f888503dafac" containerName="revoke-keys"
Jul 23 17:52:04.124919 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:52:04.124897    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5mz8v/must-gather-dz64n"
Jul 23 17:52:04.128074 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:52:04.128052    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-5mz8v\"/\"default-dockercfg-htj2r\""
Jul 23 17:52:04.128175 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:52:04.128082    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-5mz8v\"/\"kube-root-ca.crt\""
Jul 23 17:52:04.128594 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:52:04.128578    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-5mz8v\"/\"openshift-service-ca.crt\""
Jul 23 17:52:04.135200 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:52:04.135178    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-5mz8v/must-gather-dz64n"]
Jul 23 17:52:04.139148 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:52:04.139118    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:52:04.161358 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:52:04.161327    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvqmd\" (UniqueName: \"kubernetes.io/projected/bdc3e94e-8b10-465a-8894-bcec67076975-kube-api-access-bvqmd\") pod \"must-gather-dz64n\" (UID: \"bdc3e94e-8b10-465a-8894-bcec67076975\") " pod="openshift-must-gather-5mz8v/must-gather-dz64n"
Jul 23 17:52:04.161523 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:52:04.161370    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/bdc3e94e-8b10-465a-8894-bcec67076975-must-gather-output\") pod \"must-gather-dz64n\" (UID: \"bdc3e94e-8b10-465a-8894-bcec67076975\") " pod="openshift-must-gather-5mz8v/must-gather-dz64n"
Jul 23 17:52:04.235091 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:52:04.235061    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log" limit=1024
Jul 23 17:52:04.262306 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:52:04.262285    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvqmd\" (UniqueName: \"kubernetes.io/projected/bdc3e94e-8b10-465a-8894-bcec67076975-kube-api-access-bvqmd\") pod \"must-gather-dz64n\" (UID: \"bdc3e94e-8b10-465a-8894-bcec67076975\") " pod="openshift-must-gather-5mz8v/must-gather-dz64n"
Jul 23 17:52:04.262403 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:52:04.262325    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/bdc3e94e-8b10-465a-8894-bcec67076975-must-gather-output\") pod \"must-gather-dz64n\" (UID: \"bdc3e94e-8b10-465a-8894-bcec67076975\") " pod="openshift-must-gather-5mz8v/must-gather-dz64n"
Jul 23 17:52:04.262659 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:52:04.262644    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/bdc3e94e-8b10-465a-8894-bcec67076975-must-gather-output\") pod \"must-gather-dz64n\" (UID: \"bdc3e94e-8b10-465a-8894-bcec67076975\") " pod="openshift-must-gather-5mz8v/must-gather-dz64n"
Jul 23 17:52:04.283109 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:52:04.283080    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvqmd\" (UniqueName: \"kubernetes.io/projected/bdc3e94e-8b10-465a-8894-bcec67076975-kube-api-access-bvqmd\") pod \"must-gather-dz64n\" (UID: \"bdc3e94e-8b10-465a-8894-bcec67076975\") " pod="openshift-must-gather-5mz8v/must-gather-dz64n"
Jul 23 17:52:04.433945 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:52:04.433867    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5mz8v/must-gather-dz64n"
Jul 23 17:52:04.572234 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:52:04.572210    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-5mz8v/must-gather-dz64n"]
Jul 23 17:52:04.573885 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:52:04.573855    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbdc3e94e_8b10_465a_8894_bcec67076975.slice/crio-03daf8600d0f05880d0449cf74c4fbca14ca9588d56e2c61511b0b324d27337b WatchSource:0}: Error finding container 03daf8600d0f05880d0449cf74c4fbca14ca9588d56e2c61511b0b324d27337b: Status 404 returned error can't find the container with id 03daf8600d0f05880d0449cf74c4fbca14ca9588d56e2c61511b0b324d27337b
Jul 23 17:52:05.182784 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:52:05.182751    2577 scope.go:117] "RemoveContainer" containerID="783cf0c85dff80a48f68d488b05a0b4d868a6663b08ebf51dd555672d7210a69"
Jul 23 17:52:05.191494 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:52:05.191471    2577 scope.go:117] "RemoveContainer" containerID="c06d592d5414d7d37b9ed5e83972f4334db60dca8a865f57e8b16469754156fe"
Jul 23 17:52:05.368785 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:52:05.368744    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5mz8v/must-gather-dz64n" event={"ID":"bdc3e94e-8b10-465a-8894-bcec67076975","Type":"ContainerStarted","Data":"03daf8600d0f05880d0449cf74c4fbca14ca9588d56e2c61511b0b324d27337b"}
Jul 23 17:52:06.379749 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:52:06.379712    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5mz8v/must-gather-dz64n" event={"ID":"bdc3e94e-8b10-465a-8894-bcec67076975","Type":"ContainerStarted","Data":"098aeb07c2c6aaa93189c68199138de81cd887c6c24b86e8e0a5771ab993a8b1"}
Jul 23 17:52:06.380232 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:52:06.379759    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5mz8v/must-gather-dz64n" event={"ID":"bdc3e94e-8b10-465a-8894-bcec67076975","Type":"ContainerStarted","Data":"dc875436cf097415ab1eeb8e4033ace9dc0825ed29f044c02d5a751bd38ece96"}
Jul 23 17:52:06.396707 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:52:06.396645    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-5mz8v/must-gather-dz64n" podStartSLOduration=1.144396993 podStartE2EDuration="2.396623178s" totalImagesPullingTime="1.252226185s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 17:52:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 17:52:06.395404092 +0000 UTC m=+2822.165689639" watchObservedRunningTime="2026-07-23 17:52:06.396623178 +0000 UTC m=+2822.166908698"
Jul 23 17:52:07.535800 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:52:07.535772    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-5g97l_aeb10e6f-ad01-4a83-8090-049d9e919696/global-pull-secret-syncer/0.log"
Jul 23 17:52:07.694840 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:52:07.694788    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-pc8xz_9a44608a-7a55-45f6-a4dc-199578a1d620/konnectivity-agent/0.log"
Jul 23 17:52:07.797850 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:52:07.797765    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-129-216.ec2.internal_7bd1d5c1fbf257c4a5dcb5c0b12f7f75/haproxy/0.log"
Jul 23 17:52:11.730195 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:52:11.730139    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759zx982_67c1a8e8-ea9e-4d80-b4e8-cecbb603a745/extract/0.log"
Jul 23 17:52:11.768060 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:52:11.767976    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759zx982_67c1a8e8-ea9e-4d80-b4e8-cecbb603a745/util/0.log"
Jul 23 17:52:11.798488 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:52:11.798430    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759zx982_67c1a8e8-ea9e-4d80-b4e8-cecbb603a745/pull/0.log"
Jul 23 17:52:11.846210 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:52:11.846181    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xpmjs_62a7ca3d-ef60-422c-bef3-627044060c5f/extract/0.log"
Jul 23 17:52:11.878714 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:52:11.878687    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xpmjs_62a7ca3d-ef60-422c-bef3-627044060c5f/util/0.log"
Jul 23 17:52:11.913378 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:52:11.913349    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0xpmjs_62a7ca3d-ef60-422c-bef3-627044060c5f/pull/0.log"
Jul 23 17:52:11.950185 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:52:11.950144    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xchhq_283546e1-df7c-49a0-a72a-2318766fc257/extract/0.log"
Jul 23 17:52:11.978703 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:52:11.978678    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xchhq_283546e1-df7c-49a0-a72a-2318766fc257/util/0.log"
Jul 23 17:52:12.004940 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:52:12.004906    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73xchhq_283546e1-df7c-49a0-a72a-2318766fc257/pull/0.log"
Jul 23 17:52:12.039807 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:52:12.039677    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jqkqj_f0f53fe8-060d-4ddc-94a6-c361cc5d8688/extract/0.log"
Jul 23 17:52:12.084459 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:52:12.084426    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jqkqj_f0f53fe8-060d-4ddc-94a6-c361cc5d8688/util/0.log"
Jul 23 17:52:12.119869 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:52:12.119835    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jqkqj_f0f53fe8-060d-4ddc-94a6-c361cc5d8688/pull/0.log"
Jul 23 17:52:12.523651 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:52:12.523615    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-qtrml_35d67fdc-c2d6-4774-9912-1e088f3487bd/manager/0.log"
Jul 23 17:52:12.631279 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:52:12.630420    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-bd4jn_d572c939-0f51-4f05-950c-fe8ce11eacd3/registry-server/0.log"
Jul 23 17:52:12.852322 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:52:12.852132    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-hxc6t_dbd9fff4-9aef-4496-b026-e7cc838cd76f/manager/0.log"
Jul 23 17:52:12.966712 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:52:12.964632    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-pbmw7_f702fd02-0777-4c66-acf3-2463cd91eafe/manager/0.log"
Jul 23 17:52:14.730567 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:52:14.730501    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_1f6e4496-99c7-4960-96d8-e1a50c52ac7c/alertmanager/0.log"
Jul 23 17:52:14.780063 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:52:14.780030    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_1f6e4496-99c7-4960-96d8-e1a50c52ac7c/config-reloader/0.log"
Jul 23 17:52:14.814605 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:52:14.814578    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_1f6e4496-99c7-4960-96d8-e1a50c52ac7c/kube-rbac-proxy-web/0.log"
Jul 23 17:52:14.841043 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:52:14.841018    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_1f6e4496-99c7-4960-96d8-e1a50c52ac7c/kube-rbac-proxy/0.log"
Jul 23 17:52:14.874811 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:52:14.874783    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_1f6e4496-99c7-4960-96d8-e1a50c52ac7c/kube-rbac-proxy-metric/0.log"
Jul 23 17:52:14.902327 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:52:14.902290    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_1f6e4496-99c7-4960-96d8-e1a50c52ac7c/prom-label-proxy/0.log"
Jul 23 17:52:14.932167 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:52:14.932129    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_1f6e4496-99c7-4960-96d8-e1a50c52ac7c/init-config-reloader/0.log"
Jul 23 17:52:15.128539 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:52:15.128434    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-69d5f9b95d-ptst2_33724e8d-1c37-43f7-bafe-01809d1b90ed/metrics-server/0.log"
Jul 23 17:52:15.173197 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:52:15.173170    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-7b7b7bc8bf-b9gk6_3530bb86-ef19-4c9c-a150-920b342ad3e7/monitoring-plugin/0.log"
Jul 23 17:52:15.227930 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:52:15.227876    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-7qzvq_48493013-e9c4-4bb6-a9e2-060f45ad2f9c/node-exporter/0.log"
Jul 23 17:52:15.258148 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:52:15.258115    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-7qzvq_48493013-e9c4-4bb6-a9e2-060f45ad2f9c/kube-rbac-proxy/0.log"
Jul 23 17:52:15.288396 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:52:15.288368    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-7qzvq_48493013-e9c4-4bb6-a9e2-060f45ad2f9c/init-textfile/0.log"
Jul 23 17:52:15.535614 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:52:15.535579    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-79f6958cdf-c2jxp_c3cfd256-a951-47aa-a1ff-2604c290b07d/kube-rbac-proxy-main/0.log"
Jul 23 17:52:15.562825 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:52:15.562796    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-79f6958cdf-c2jxp_c3cfd256-a951-47aa-a1ff-2604c290b07d/kube-rbac-proxy-self/0.log"
Jul 23 17:52:15.589721 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:52:15.589693    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-79f6958cdf-c2jxp_c3cfd256-a951-47aa-a1ff-2604c290b07d/openshift-state-metrics/0.log"
Jul 23 17:52:15.923205 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:52:15.923121    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-5945b5cfd-c8579_3d3ac6d1-2506-4b3e-9336-4f8bdc1a87c2/prometheus-operator/0.log"
Jul 23 17:52:15.945263 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:52:15.945232    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-5945b5cfd-c8579_3d3ac6d1-2506-4b3e-9336-4f8bdc1a87c2/kube-rbac-proxy/0.log"
Jul 23 17:52:16.005432 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:52:16.005405    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-56745b4487-g2wzx_d128b7fd-71ee-4da9-8d06-bdcb68513773/telemeter-client/0.log"
Jul 23 17:52:16.046480 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:52:16.046456    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-56745b4487-g2wzx_d128b7fd-71ee-4da9-8d06-bdcb68513773/reload/0.log"
Jul 23 17:52:16.070928 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:52:16.070893    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-56745b4487-g2wzx_d128b7fd-71ee-4da9-8d06-bdcb68513773/kube-rbac-proxy/0.log"
Jul 23 17:52:16.105654 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:52:16.105627    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5ff84d65c6-d2g5h_f50cede5-05ed-439c-b3f1-f2fa3c364d79/thanos-query/0.log"
Jul 23 17:52:16.130647 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:52:16.130618    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5ff84d65c6-d2g5h_f50cede5-05ed-439c-b3f1-f2fa3c364d79/kube-rbac-proxy-web/0.log"
Jul 23 17:52:16.162085 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:52:16.162049    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5ff84d65c6-d2g5h_f50cede5-05ed-439c-b3f1-f2fa3c364d79/kube-rbac-proxy/0.log"
Jul 23 17:52:16.186830 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:52:16.186756    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5ff84d65c6-d2g5h_f50cede5-05ed-439c-b3f1-f2fa3c364d79/prom-label-proxy/0.log"
Jul 23 17:52:16.212517 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:52:16.212490    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5ff84d65c6-d2g5h_f50cede5-05ed-439c-b3f1-f2fa3c364d79/kube-rbac-proxy-rules/0.log"
Jul 23 17:52:16.236936 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:52:16.236906    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5ff84d65c6-d2g5h_f50cede5-05ed-439c-b3f1-f2fa3c364d79/kube-rbac-proxy-metrics/0.log"
Jul 23 17:52:17.464077 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:52:17.464037    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-5mz8v/perf-node-gather-daemonset-qpb2d"]
Jul 23 17:52:17.467968 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:52:17.467933    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5mz8v/perf-node-gather-daemonset-qpb2d"
Jul 23 17:52:17.482203 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:52:17.482174    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-5mz8v/perf-node-gather-daemonset-qpb2d"]
Jul 23 17:52:17.501090 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:52:17.500755    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/77d6f5fc-14ab-4ade-97f4-4d6598f3f2a4-sys\") pod \"perf-node-gather-daemonset-qpb2d\" (UID: \"77d6f5fc-14ab-4ade-97f4-4d6598f3f2a4\") " pod="openshift-must-gather-5mz8v/perf-node-gather-daemonset-qpb2d"
Jul 23 17:52:17.501090 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:52:17.500810    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/77d6f5fc-14ab-4ade-97f4-4d6598f3f2a4-proc\") pod \"perf-node-gather-daemonset-qpb2d\" (UID: \"77d6f5fc-14ab-4ade-97f4-4d6598f3f2a4\") " pod="openshift-must-gather-5mz8v/perf-node-gather-daemonset-qpb2d"
Jul 23 17:52:17.501090 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:52:17.500839    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvlbp\" (UniqueName: \"kubernetes.io/projected/77d6f5fc-14ab-4ade-97f4-4d6598f3f2a4-kube-api-access-pvlbp\") pod \"perf-node-gather-daemonset-qpb2d\" (UID: \"77d6f5fc-14ab-4ade-97f4-4d6598f3f2a4\") " pod="openshift-must-gather-5mz8v/perf-node-gather-daemonset-qpb2d"
Jul 23 17:52:17.501090 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:52:17.500884    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/77d6f5fc-14ab-4ade-97f4-4d6598f3f2a4-lib-modules\") pod \"perf-node-gather-daemonset-qpb2d\" (UID: \"77d6f5fc-14ab-4ade-97f4-4d6598f3f2a4\") " pod="openshift-must-gather-5mz8v/perf-node-gather-daemonset-qpb2d"
Jul 23 17:52:17.501090 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:52:17.500940    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/77d6f5fc-14ab-4ade-97f4-4d6598f3f2a4-podres\") pod \"perf-node-gather-daemonset-qpb2d\" (UID: \"77d6f5fc-14ab-4ade-97f4-4d6598f3f2a4\") " pod="openshift-must-gather-5mz8v/perf-node-gather-daemonset-qpb2d"
Jul 23 17:52:17.601780 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:52:17.601732    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/77d6f5fc-14ab-4ade-97f4-4d6598f3f2a4-podres\") pod \"perf-node-gather-daemonset-qpb2d\" (UID: \"77d6f5fc-14ab-4ade-97f4-4d6598f3f2a4\") " pod="openshift-must-gather-5mz8v/perf-node-gather-daemonset-qpb2d"
Jul 23 17:52:17.601980 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:52:17.601855    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/77d6f5fc-14ab-4ade-97f4-4d6598f3f2a4-sys\") pod \"perf-node-gather-daemonset-qpb2d\" (UID: \"77d6f5fc-14ab-4ade-97f4-4d6598f3f2a4\") " pod="openshift-must-gather-5mz8v/perf-node-gather-daemonset-qpb2d"
Jul 23 17:52:17.601980 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:52:17.601892    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/77d6f5fc-14ab-4ade-97f4-4d6598f3f2a4-proc\") pod \"perf-node-gather-daemonset-qpb2d\" (UID: \"77d6f5fc-14ab-4ade-97f4-4d6598f3f2a4\") " pod="openshift-must-gather-5mz8v/perf-node-gather-daemonset-qpb2d"
Jul 23 17:52:17.601980 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:52:17.601919    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvlbp\" (UniqueName: \"kubernetes.io/projected/77d6f5fc-14ab-4ade-97f4-4d6598f3f2a4-kube-api-access-pvlbp\") pod \"perf-node-gather-daemonset-qpb2d\" (UID: \"77d6f5fc-14ab-4ade-97f4-4d6598f3f2a4\") " pod="openshift-must-gather-5mz8v/perf-node-gather-daemonset-qpb2d"
Jul 23 17:52:17.601980 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:52:17.601965    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/77d6f5fc-14ab-4ade-97f4-4d6598f3f2a4-lib-modules\") pod \"perf-node-gather-daemonset-qpb2d\" (UID: \"77d6f5fc-14ab-4ade-97f4-4d6598f3f2a4\") " pod="openshift-must-gather-5mz8v/perf-node-gather-daemonset-qpb2d"
Jul 23 17:52:17.602221 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:52:17.602155    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/77d6f5fc-14ab-4ade-97f4-4d6598f3f2a4-lib-modules\") pod \"perf-node-gather-daemonset-qpb2d\" (UID: \"77d6f5fc-14ab-4ade-97f4-4d6598f3f2a4\") " pod="openshift-must-gather-5mz8v/perf-node-gather-daemonset-qpb2d"
Jul 23 17:52:17.602299 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:52:17.602275    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/77d6f5fc-14ab-4ade-97f4-4d6598f3f2a4-podres\") pod \"perf-node-gather-daemonset-qpb2d\" (UID: \"77d6f5fc-14ab-4ade-97f4-4d6598f3f2a4\") " pod="openshift-must-gather-5mz8v/perf-node-gather-daemonset-qpb2d"
Jul 23 17:52:17.602369 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:52:17.602333    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/77d6f5fc-14ab-4ade-97f4-4d6598f3f2a4-sys\") pod \"perf-node-gather-daemonset-qpb2d\" (UID: \"77d6f5fc-14ab-4ade-97f4-4d6598f3f2a4\") " pod="openshift-must-gather-5mz8v/perf-node-gather-daemonset-qpb2d"
Jul 23 17:52:17.602426 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:52:17.602379    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/77d6f5fc-14ab-4ade-97f4-4d6598f3f2a4-proc\") pod \"perf-node-gather-daemonset-qpb2d\" (UID: \"77d6f5fc-14ab-4ade-97f4-4d6598f3f2a4\") " pod="openshift-must-gather-5mz8v/perf-node-gather-daemonset-qpb2d"
Jul 23 17:52:17.613684 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:52:17.613660    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvlbp\" (UniqueName: \"kubernetes.io/projected/77d6f5fc-14ab-4ade-97f4-4d6598f3f2a4-kube-api-access-pvlbp\") pod \"perf-node-gather-daemonset-qpb2d\" (UID: \"77d6f5fc-14ab-4ade-97f4-4d6598f3f2a4\") " pod="openshift-must-gather-5mz8v/perf-node-gather-daemonset-qpb2d"
Jul 23 17:52:17.786580 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:52:17.786491    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5mz8v/perf-node-gather-daemonset-qpb2d"
Jul 23 17:52:17.982233 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:52:17.982181    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-5mz8v/perf-node-gather-daemonset-qpb2d"]
Jul 23 17:52:17.983911 ip-10-0-129-216 kubenswrapper[2577]: W0723 17:52:17.983874    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod77d6f5fc_14ab_4ade_97f4_4d6598f3f2a4.slice/crio-627c045dd8e1091b798b4baffccc1bfe39113d9de3effb20d5458ed496d59515 WatchSource:0}: Error finding container 627c045dd8e1091b798b4baffccc1bfe39113d9de3effb20d5458ed496d59515: Status 404 returned error can't find the container with id 627c045dd8e1091b798b4baffccc1bfe39113d9de3effb20d5458ed496d59515
Jul 23 17:52:18.506539 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:52:18.506509    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5mz8v/perf-node-gather-daemonset-qpb2d" event={"ID":"77d6f5fc-14ab-4ade-97f4-4d6598f3f2a4","Type":"ContainerStarted","Data":"d532decc32c7376642fc4adef96ce5056e85ac34a6a17e0cd361edf4d7da1475"}
Jul 23 17:52:18.507021 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:52:18.506563    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5mz8v/perf-node-gather-daemonset-qpb2d" event={"ID":"77d6f5fc-14ab-4ade-97f4-4d6598f3f2a4","Type":"ContainerStarted","Data":"627c045dd8e1091b798b4baffccc1bfe39113d9de3effb20d5458ed496d59515"}
Jul 23 17:52:18.507021 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:52:18.506652    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-5mz8v/perf-node-gather-daemonset-qpb2d"
Jul 23 17:52:18.523656 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:52:18.523606    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-5mz8v/perf-node-gather-daemonset-qpb2d" podStartSLOduration=1.523590285 podStartE2EDuration="1.523590285s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 17:52:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 17:52:18.522571431 +0000 UTC m=+2834.292856952" watchObservedRunningTime="2026-07-23 17:52:18.523590285 +0000 UTC m=+2834.293875804"
Jul 23 17:52:20.632584 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:52:20.632541    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-9pw6q_e3a03180-5756-44d1-adb3-5844e27389ff/dns/0.log"
Jul 23 17:52:20.659885 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:52:20.659860    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-9pw6q_e3a03180-5756-44d1-adb3-5844e27389ff/kube-rbac-proxy/0.log"
Jul 23 17:52:20.841389 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:52:20.841359    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-stw6g_5c929500-877b-43d7-9710-0ea27db45055/dns-node-resolver/0.log"
Jul 23 17:52:21.576629 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:52:21.576601    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-znmk5_564dda72-0cfc-4e30-b04d-eddd1a9b7b30/node-ca/0.log"
Jul 23 17:52:22.939955 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:52:22.939921    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5bf47cd988-tb75s_7f87b7f7-f222-43da-9f48-37f88ae96fc5/payload-processing/0.log"
Jul 23 17:52:23.745318 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:52:23.745212    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-gqsnn_1f32f423-161d-444d-9698-040a50020b5c/serve-healthcheck-canary/0.log"
Jul 23 17:52:24.487305 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:52:24.487276    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-89shm_0372b049-248a-43e5-84a8-198ec9bbc1cd/kube-rbac-proxy/0.log"
Jul 23 17:52:24.510038 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:52:24.510009    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-89shm_0372b049-248a-43e5-84a8-198ec9bbc1cd/exporter/0.log"
Jul 23 17:52:24.527155 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:52:24.527123    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-5mz8v/perf-node-gather-daemonset-qpb2d"
Jul 23 17:52:24.536100 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:52:24.536058    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-89shm_0372b049-248a-43e5-84a8-198ec9bbc1cd/extractor/0.log"
Jul 23 17:52:27.191990 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:52:27.189844    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-6svpm_4e7eff36-6da5-43e5-be32-820f601c91c7/manager/0.log"
Jul 23 17:52:27.348965 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:52:27.348930    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7979b9b8c-vm92w_3f763d30-b0dc-4dd9-b0b1-b91ee5c631f9/manager/0.log"
Jul 23 17:52:27.479661 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:52:27.479620    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-z4mjc_aa92e815-3d31-4472-b7ef-ec1f76756bce/manager/0.log"
Jul 23 17:52:27.581304 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:52:27.580975    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-f9b469455-874ln_317747fa-9885-43a3-a464-bbdb37ac9a6c/manager/0.log"
Jul 23 17:52:27.602424 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:52:27.602397    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-f9b469455-874ln_317747fa-9885-43a3-a464-bbdb37ac9a6c/copy-manifests/0.log"
Jul 23 17:52:28.078009 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:52:28.077983    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5bf47cd988-tb75s_7f87b7f7-f222-43da-9f48-37f88ae96fc5/payload-processing/0.log"
Jul 23 17:52:29.033286 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:52:29.033227    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-65fcfccbcc-2stkg_dedd689a-b86a-4b58-81e0-93cbd069dcb9/manager/0.log"
Jul 23 17:52:29.649163 ip-10-0-129-216 kubenswrapper[2577]: I0723 17:52:29.649128    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5bf47cd988-tb75s_7f87b7f7-f222-43da-9f48-37f88ae96fc5/payload-processing/0.log"