Jul 07 15:40:59.587091 ip-10-0-132-65 systemd[1]: Starting Kubernetes Kubelet... Jul 07 15:41:00.062217 ip-10-0-132-65 kubenswrapper[2565]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 07 15:41:00.062217 ip-10-0-132-65 kubenswrapper[2565]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 07 15:41:00.062217 ip-10-0-132-65 kubenswrapper[2565]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 07 15:41:00.062217 ip-10-0-132-65 kubenswrapper[2565]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 07 15:41:00.062217 ip-10-0-132-65 kubenswrapper[2565]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 07 15:41:00.062217 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.030639 2565 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 07 15:41:00.062217 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.035187 2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 07 15:41:00.062217 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.035209 2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 07 15:41:00.062217 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.035213 2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 07 15:41:00.062217 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.035217 2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 07 15:41:00.062217 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.035220 2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 07 15:41:00.062217 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.035224 2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 07 15:41:00.062217 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.035227 2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 07 15:41:00.062217 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.035229 2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 07 15:41:00.064293 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.035232 2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 07 15:41:00.064293 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.035235 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 07 15:41:00.064293 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.035238 2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 07 15:41:00.064293 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.035241 2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 07 15:41:00.064293 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.035243 2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 07 15:41:00.064293 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.035254 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 07 15:41:00.064293 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.035257 2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 07 15:41:00.064293 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.035259 2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 07 15:41:00.064293 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.035262 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 07 15:41:00.064293 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.035265 2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 07 15:41:00.064293 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.035268 2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 07 15:41:00.064293 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.035272 2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 07 15:41:00.064293 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.035275 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 07 15:41:00.064293 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.035278 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 07 15:41:00.064293 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.035281 2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 07 15:41:00.064293 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.035284 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 07 15:41:00.064293 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.035294 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 07 15:41:00.064293 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.035298 2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 07 15:41:00.064293 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.035301 2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 07 15:41:00.064293 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.035303 2565 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 07 15:41:00.077050 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.035306 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 07 15:41:00.077050 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.035309 2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 07 15:41:00.077050 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.035311 2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 07 15:41:00.077050 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.035314 2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 07 15:41:00.077050 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.035317 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 07 15:41:00.077050 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.035319 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 07 15:41:00.077050 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.035322 2565 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 07 15:41:00.077050 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.035324 2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 07 15:41:00.077050 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.035327 2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 07 15:41:00.077050 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.035329 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 07 15:41:00.077050 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.035332 2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 07 15:41:00.077050 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.035334 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 07 15:41:00.077050 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.035337 2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 07 15:41:00.077050 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.035340 2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 07 15:41:00.077050 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.035342 2565 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 07 15:41:00.077050 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.035345 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 07 15:41:00.077050 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.035348 2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 07 15:41:00.077050 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.035351 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 07 15:41:00.077050 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.035354 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 07 15:41:00.078960 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.035357 2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 07 15:41:00.078960 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.035361 2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 07 15:41:00.078960 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.035363 2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 07 15:41:00.078960 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.035366 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 07 15:41:00.078960 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.035369 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 07 15:41:00.078960 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.035372 2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 07 15:41:00.078960 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.035374 2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 07 15:41:00.078960 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.035377 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 07 15:41:00.078960 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.035379 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 07 15:41:00.078960 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.035382 2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 07 15:41:00.078960 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.035384 2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 07 15:41:00.078960 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.035387 2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 07 15:41:00.078960 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.035390 2565 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 07 15:41:00.078960 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.035392 2565 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 07 15:41:00.078960 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.035395 2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 07 15:41:00.078960 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.035397 2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 07 15:41:00.078960 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.035400 2565 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 07 15:41:00.078960 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.035403 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 07 15:41:00.078960 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.035405 2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 07 15:41:00.078960 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.035408 2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 07 15:41:00.081238 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.035410 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 07 15:41:00.081238 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.035415 2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 07 15:41:00.081238 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.035418 2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 07 15:41:00.081238 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.035423 2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 07 15:41:00.081238 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.035426 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 07 15:41:00.081238 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.035429 2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 07 15:41:00.081238 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.035432 2565 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 07 15:41:00.081238 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.035435 2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 07 15:41:00.081238 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.035438 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 07 15:41:00.081238 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.035441 2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 07 15:41:00.081238 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.035443 2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 07 15:41:00.081238 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.035462 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 07 15:41:00.081238 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.035471 2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 07 15:41:00.081238 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.035477 2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 07 15:41:00.081238 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.035482 2565 feature_gate.go:328] unrecognized feature gate: Example Jul 07 15:41:00.081238 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.035485 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 07 15:41:00.081238 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.035489 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 07 15:41:00.081238 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.035491 2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 07 15:41:00.096624 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.035494 2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 07 15:41:00.096624 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.035497 2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 07 15:41:00.096624 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.035500 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 07 15:41:00.096624 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.035502 2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 07 15:41:00.096624 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.035505 2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 07 15:41:00.096624 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.035508 2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 07 15:41:00.096624 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.035511 2565 feature_gate.go:328] unrecognized feature gate: Example2 Jul 07 15:41:00.096624 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.035513 2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 07 15:41:00.096624 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.035516 2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 07 15:41:00.096624 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.035519 2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 07 15:41:00.096624 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.035521 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 07 15:41:00.096624 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.035524 2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 07 15:41:00.096624 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.035526 2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 07 15:41:00.096624 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.035530 2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 07 15:41:00.096624 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.036268 2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 07 15:41:00.096624 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.036275 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 07 15:41:00.096624 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.036279 2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 07 15:41:00.096624 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.036282 2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 07 15:41:00.096624 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.036286 2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 07 15:41:00.096624 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.036288 2565 feature_gate.go:328] unrecognized feature gate: Example Jul 07 15:41:00.099441 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.036291 2565 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 07 15:41:00.099441 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.036294 2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 07 15:41:00.099441 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.036297 2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 07 15:41:00.099441 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.036300 2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 07 15:41:00.099441 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.036303 2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 07 15:41:00.099441 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.036306 2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 07 15:41:00.099441 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.036309 2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 07 15:41:00.099441 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.036313 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 07 15:41:00.099441 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.036315 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 07 15:41:00.099441 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.036318 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 07 15:41:00.099441 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.036321 2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 07 15:41:00.099441 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.036323 2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 07 15:41:00.099441 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.036326 2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 07 15:41:00.099441 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.036329 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 07 15:41:00.099441 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.036331 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 07 15:41:00.099441 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.036334 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 07 15:41:00.099441 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.036337 2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 07 15:41:00.099441 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.036341 2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 07 15:41:00.099441 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.036346 2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 07 15:41:00.101541 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.036348 2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 07 15:41:00.101541 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.036351 2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 07 15:41:00.101541 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.036354 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 07 15:41:00.101541 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.036356 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 07 15:41:00.101541 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.036359 2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 07 15:41:00.101541 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.036361 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 07 15:41:00.101541 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.036364 2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 07 15:41:00.101541 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.036367 2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 07 15:41:00.101541 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.036369 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 07 15:41:00.101541 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.036372 2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 07 15:41:00.101541 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.036375 2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 07 15:41:00.101541 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.036378 2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 07 15:41:00.101541 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.036380 2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 07 15:41:00.101541 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.036383 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 07 15:41:00.101541 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.036386 2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 07 15:41:00.101541 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.036388 2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 07 15:41:00.101541 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.036391 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 07 15:41:00.101541 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.036394 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 07 15:41:00.101541 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.036396 2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 07 15:41:00.101541 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.036399 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 07 15:41:00.103849 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.036401 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 07 15:41:00.103849 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.036405 2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 07 15:41:00.103849 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.036407 2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 07 15:41:00.103849 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.036410 2565 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 07 15:41:00.103849 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.036413 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 07 15:41:00.103849 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.036416 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 07 15:41:00.103849 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.036418 2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 07 15:41:00.103849 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.036421 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 07 15:41:00.103849 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.036424 2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 07 15:41:00.103849 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.036429 2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 07 15:41:00.103849 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.036435 2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 07 15:41:00.103849 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.036439 2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 07 15:41:00.103849 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.036442 2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 07 15:41:00.103849 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.036445 2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 07 15:41:00.103849 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.036447 2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 07 15:41:00.103849 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.036450 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 07 15:41:00.103849 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.036452 2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 07 15:41:00.103849 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.036455 2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 07 15:41:00.103849 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.036457 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 07 15:41:00.116603 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.036460 2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 07 15:41:00.116603 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.036463 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 07 15:41:00.116603 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.036465 2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 07 15:41:00.116603 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.036468 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 07 15:41:00.116603 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.036471 2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 07 15:41:00.116603 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.036473 2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 07 15:41:00.116603 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.036477 2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 07 15:41:00.116603 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.036480 2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 07 15:41:00.116603 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.036482 2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 07 15:41:00.116603 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.036485 2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 07 15:41:00.116603 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.036487 2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 07 15:41:00.116603 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.036490 2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 07 15:41:00.116603 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.036492 2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 07 15:41:00.116603 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.036495 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 07 15:41:00.116603 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.036498 2565 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 07 15:41:00.116603 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.036501 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 07 15:41:00.116603 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.036503 2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 07 15:41:00.116603 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.036506 2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 07 15:41:00.116603 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.036508 2565 feature_gate.go:328] unrecognized feature gate: Example2 Jul 07 15:41:00.110506 ip-10-0-132-65 systemd[1]: Started Kubernetes Kubelet. Jul 07 15:41:00.119472 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.036511 2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 07 15:41:00.119472 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.036513 2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 07 15:41:00.119472 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.036516 2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 07 15:41:00.119472 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.036520 2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 07 15:41:00.119472 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.036523 2565 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 07 15:41:00.119472 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.036526 2565 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 07 15:41:00.119472 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.036528 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 07 15:41:00.119472 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.036531 2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 07 15:41:00.119472 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.036533 2565 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 07 15:41:00.119472 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.036536 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 07 15:41:00.119472 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.036539 2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 07 15:41:00.119472 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.036541 2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 07 15:41:00.119472 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.036544 2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 07 15:41:00.119472 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.036546 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 07 15:41:00.119472 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.036549 2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 07 15:41:00.119472 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.036551 2565 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 07 15:41:00.119472 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.036950 2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 07 15:41:00.119472 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.036956 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 07 15:41:00.119472 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.036959 2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 07 15:41:00.119472 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.036962 2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 07 15:41:00.132985 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.036965 2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 07 15:41:00.132985 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.036968 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 07 15:41:00.132985 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.036971 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 07 15:41:00.132985 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.036974 2565 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 07 15:41:00.132985 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.036977 2565 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 07 15:41:00.132985 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.036981 2565 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 07 15:41:00.132985 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.036984 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 07 15:41:00.132985 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.036988 2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 07 15:41:00.132985 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.036992 2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 07 15:41:00.132985 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.036995 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 07 15:41:00.132985 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.036998 2565 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 07 15:41:00.132985 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.037002 2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 07 15:41:00.132985 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.037004 2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 07 15:41:00.132985 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.037007 2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 07 15:41:00.132985 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.037011 2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 07 15:41:00.132985 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.037016 2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 07 15:41:00.132985 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.037020 2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 07 15:41:00.132985 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.037023 2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 07 15:41:00.132985 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.037026 2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 07 15:41:00.132985 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.037029 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 07 15:41:00.149053 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.037031 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 07 15:41:00.149053 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.037034 2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 07 15:41:00.149053 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.037036 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 07 15:41:00.149053 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.037039 2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 07 15:41:00.149053 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.037042 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 07 15:41:00.149053 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.037044 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 07 15:41:00.149053 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.037047 2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 07 15:41:00.149053 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.037050 2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 07 15:41:00.149053 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.037052 2565 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 07 15:41:00.149053 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.037055 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 07 15:41:00.149053 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.037057 2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 07 15:41:00.149053 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.037060 2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 07 15:41:00.149053 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.037063 2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 07 15:41:00.149053 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.037065 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 07 15:41:00.149053 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.037068 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 07 15:41:00.149053 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.037070 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 07 15:41:00.149053 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.037073 2565 feature_gate.go:328] unrecognized feature gate: Example2 Jul 07 15:41:00.149053 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.037077 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 07 15:41:00.149053 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.037080 2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 07 15:41:00.149053 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.037082 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 07 15:41:00.164704 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.037085 2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 07 15:41:00.164704 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.037088 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 07 15:41:00.164704 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.037090 2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 07 15:41:00.164704 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.037093 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 07 15:41:00.164704 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.037095 2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 07 15:41:00.164704 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.037098 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 07 15:41:00.164704 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.037100 2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 07 15:41:00.164704 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.037104 2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 07 15:41:00.164704 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.037106 2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 07 15:41:00.164704 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.037109 2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 07 15:41:00.164704 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.037111 2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 07 15:41:00.164704 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.037114 2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 07 15:41:00.164704 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.037117 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 07 15:41:00.164704 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.037119 2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 07 15:41:00.164704 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.037121 2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 07 15:41:00.164704 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.037124 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 07 15:41:00.164704 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.037127 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 07 15:41:00.164704 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.037129 2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 07 15:41:00.164704 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.037132 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 07 15:41:00.167548 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.037134 2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 07 15:41:00.167548 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.037137 2565 feature_gate.go:328] unrecognized feature gate: Example Jul 07 15:41:00.167548 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.037139 2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 07 15:41:00.167548 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.037142 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 07 15:41:00.167548 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.037144 2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 07 15:41:00.167548 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.037147 2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 07 15:41:00.167548 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.037149 2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 07 15:41:00.167548 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.037152 2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 07 15:41:00.167548 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.037155 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 07 15:41:00.167548 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.037157 2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 07 15:41:00.167548 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.037161 2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 07 15:41:00.167548 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.037163 2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 07 15:41:00.167548 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.037166 2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 07 15:41:00.167548 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.037169 2565 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 07 15:41:00.167548 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.037172 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 07 15:41:00.167548 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.037174 2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 07 15:41:00.167548 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.037177 2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 07 15:41:00.167548 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.037179 2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 07 15:41:00.167548 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.037182 2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 07 15:41:00.167548 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.037184 2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 07 15:41:00.173151 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.037190 2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 07 15:41:00.173151 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.037193 2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 07 15:41:00.173151 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.037196 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 07 15:41:00.173151 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.037199 2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 07 15:41:00.173151 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.037202 2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 07 15:41:00.173151 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.037205 2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 07 15:41:00.173151 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.037207 2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 07 15:41:00.173151 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.037210 2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 07 15:41:00.173151 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.037213 2565 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 07 15:41:00.173151 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.037216 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 07 15:41:00.173151 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.037218 2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 07 15:41:00.173151 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.037221 2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 07 15:41:00.173151 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.037223 2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 07 15:41:00.173151 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.037226 2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 07 15:41:00.173151 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.037229 2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 07 15:41:00.173151 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.037231 2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 07 15:41:00.173151 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.037300 2565 flags.go:64] FLAG: --address="0.0.0.0" Jul 07 15:41:00.173151 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.037308 2565 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 07 15:41:00.173151 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.037316 2565 flags.go:64] FLAG: --anonymous-auth="true" Jul 07 15:41:00.173151 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.037322 2565 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 07 15:41:00.173151 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.037328 2565 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 07 15:41:00.359746 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.037334 2565 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 07 15:41:00.359746 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.037342 2565 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 07 15:41:00.359746 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.037347 2565 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 07 15:41:00.359746 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.037350 2565 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 07 15:41:00.359746 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.037354 2565 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 07 15:41:00.359746 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.037357 2565 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 07 15:41:00.359746 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.037360 2565 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 07 15:41:00.359746 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.037364 2565 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 07 15:41:00.359746 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.037367 2565 flags.go:64] FLAG: --cgroup-root="" Jul 07 15:41:00.359746 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.037371 2565 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 07 15:41:00.359746 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.037373 2565 flags.go:64] FLAG: --client-ca-file="" Jul 07 15:41:00.359746 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.037377 2565 flags.go:64] FLAG: --cloud-provider="external" Jul 07 15:41:00.359746 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.037380 2565 flags.go:64] FLAG: --cluster-dns="[]" Jul 07 15:41:00.359746 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.037385 2565 flags.go:64] FLAG: --cluster-domain="" Jul 07 15:41:00.359746 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.037388 2565 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 07 15:41:00.359746 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.037391 2565 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 07 15:41:00.359746 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.037394 2565 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 07 15:41:00.359746 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.037397 2565 flags.go:64] FLAG: --container-log-max-files="5" Jul 07 15:41:00.359746 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.037401 2565 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 07 15:41:00.359746 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.037404 2565 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 07 15:41:00.359746 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.037407 2565 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 07 15:41:00.359746 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.037410 2565 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 07 15:41:00.359746 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.037413 2565 flags.go:64] FLAG: --contention-profiling="false" Jul 07 15:41:00.359746 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.037416 2565 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 07 15:41:00.361294 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.037419 2565 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 07 15:41:00.361294 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.037423 2565 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 07 15:41:00.361294 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.037425 2565 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 07 15:41:00.361294 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.037430 2565 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 07 15:41:00.361294 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.037433 2565 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 07 15:41:00.361294 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.037435 2565 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 07 15:41:00.361294 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.037438 2565 flags.go:64] FLAG: --enable-load-reader="false" Jul 07 15:41:00.361294 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.037441 2565 flags.go:64] FLAG: --enable-server="true" Jul 07 15:41:00.361294 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.037444 2565 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 07 15:41:00.361294 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.037449 2565 flags.go:64] FLAG: --event-burst="100" Jul 07 15:41:00.361294 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.037452 2565 flags.go:64] FLAG: --event-qps="50" Jul 07 15:41:00.361294 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.037455 2565 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 07 15:41:00.361294 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.037458 2565 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 07 15:41:00.361294 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.037461 2565 flags.go:64] FLAG: --eviction-hard="" Jul 07 15:41:00.361294 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.037465 2565 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 07 15:41:00.361294 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.037468 2565 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 07 15:41:00.361294 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.037471 2565 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 07 15:41:00.361294 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.037474 2565 flags.go:64] FLAG: --eviction-soft="" Jul 07 15:41:00.361294 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.037477 2565 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 07 15:41:00.361294 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.037481 2565 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 07 15:41:00.361294 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.037484 2565 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 07 15:41:00.361294 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.037487 2565 flags.go:64] FLAG: --experimental-mounter-path="" Jul 07 15:41:00.361294 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.037501 2565 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 07 15:41:00.361294 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.037504 2565 flags.go:64] FLAG: --fail-swap-on="true" Jul 07 15:41:00.361294 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.037506 2565 flags.go:64] FLAG: --feature-gates="" Jul 07 15:41:00.362453 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.037515 2565 flags.go:64] FLAG: --file-check-frequency="20s" Jul 07 15:41:00.362453 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.037518 2565 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 07 15:41:00.362453 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.037521 2565 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 07 15:41:00.362453 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.037524 2565 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 07 15:41:00.362453 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.037527 2565 flags.go:64] FLAG: --healthz-port="10248" Jul 07 15:41:00.362453 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.037530 2565 flags.go:64] FLAG: --help="false" Jul 07 15:41:00.362453 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.037533 2565 flags.go:64] FLAG: --hostname-override="ip-10-0-132-65.ec2.internal" Jul 07 15:41:00.362453 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.037536 2565 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 07 15:41:00.362453 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.037539 2565 flags.go:64] FLAG: --http-check-frequency="20s" Jul 07 15:41:00.362453 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.037542 2565 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 07 15:41:00.362453 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.037546 2565 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 07 15:41:00.362453 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.037549 2565 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 07 15:41:00.362453 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.037552 2565 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 07 15:41:00.362453 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.037555 2565 flags.go:64] FLAG: --image-service-endpoint="" Jul 07 15:41:00.362453 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.037558 2565 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 07 15:41:00.362453 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.037560 2565 flags.go:64] FLAG: --kube-api-burst="100" Jul 07 15:41:00.362453 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.037564 2565 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 07 15:41:00.362453 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.037568 2565 flags.go:64] FLAG: --kube-api-qps="50" Jul 07 15:41:00.362453 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.037570 2565 flags.go:64] FLAG: --kube-reserved="" Jul 07 15:41:00.362453 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.037573 2565 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 07 15:41:00.362453 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.037576 2565 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 07 15:41:00.362453 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.037579 2565 flags.go:64] FLAG: --kubelet-cgroups="" Jul 07 15:41:00.362453 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.037582 2565 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 07 15:41:00.362453 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.037584 2565 flags.go:64] FLAG: --lock-file="" Jul 07 15:41:00.365612 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.037588 2565 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 07 15:41:00.365612 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.037591 2565 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 07 15:41:00.365612 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.037595 2565 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 07 15:41:00.365612 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.037602 2565 flags.go:64] FLAG: --log-json-split-stream="false" Jul 07 15:41:00.365612 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.037605 2565 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 07 15:41:00.365612 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.037607 2565 flags.go:64] FLAG: --log-text-split-stream="false" Jul 07 15:41:00.365612 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.037610 2565 flags.go:64] FLAG: --logging-format="text" Jul 07 15:41:00.365612 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.037613 2565 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 07 15:41:00.365612 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.037617 2565 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 07 15:41:00.365612 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.037620 2565 flags.go:64] FLAG: --manifest-url="" Jul 07 15:41:00.365612 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.037623 2565 flags.go:64] FLAG: --manifest-url-header="" Jul 07 15:41:00.365612 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.037627 2565 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 07 15:41:00.365612 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.037630 2565 flags.go:64] FLAG: --max-open-files="1000000" Jul 07 15:41:00.365612 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.037634 2565 flags.go:64] FLAG: --max-pods="110" Jul 07 15:41:00.365612 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.037637 2565 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 07 15:41:00.365612 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.037640 2565 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 07 15:41:00.365612 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.037643 2565 flags.go:64] FLAG: --memory-manager-policy="None" Jul 07 15:41:00.365612 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.037646 2565 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 07 15:41:00.365612 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.037649 2565 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 07 15:41:00.365612 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.037652 2565 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 07 15:41:00.365612 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.037655 2565 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 07 15:41:00.365612 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.037662 2565 flags.go:64] FLAG: --node-status-max-images="50" Jul 07 15:41:00.365612 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.037665 2565 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 07 15:41:00.365612 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.037668 2565 flags.go:64] FLAG: --oom-score-adj="-999" Jul 07 15:41:00.365612 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.037673 2565 flags.go:64] FLAG: --pod-cidr="" Jul 07 15:41:00.366447 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.037676 2565 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ce93f6b4197b9c62a26a3131f8057ffe9c9379c5079a4f62e5919b75151394e5" Jul 07 15:41:00.366447 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.037682 2565 flags.go:64] FLAG: --pod-manifest-path="" Jul 07 15:41:00.366447 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.037685 2565 flags.go:64] FLAG: --pod-max-pids="-1" Jul 07 15:41:00.366447 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.037688 2565 flags.go:64] FLAG: --pods-per-core="0" Jul 07 15:41:00.366447 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.037691 2565 flags.go:64] FLAG: --port="10250" Jul 07 15:41:00.366447 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.037694 2565 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 07 15:41:00.366447 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.037697 2565 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0967f69856e340fd1" Jul 07 15:41:00.366447 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.037700 2565 flags.go:64] FLAG: --qos-reserved="" Jul 07 15:41:00.366447 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.037703 2565 flags.go:64] FLAG: --read-only-port="10255" Jul 07 15:41:00.366447 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.037706 2565 flags.go:64] FLAG: --register-node="true" Jul 07 15:41:00.366447 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.037711 2565 flags.go:64] FLAG: --register-with-taints="" Jul 07 15:41:00.366447 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.037715 2565 flags.go:64] FLAG: --registry-burst="10" Jul 07 15:41:00.366447 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.037717 2565 flags.go:64] FLAG: --registry-qps="5" Jul 07 15:41:00.366447 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.037720 2565 flags.go:64] FLAG: --reserved-cpus="" Jul 07 15:41:00.366447 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.037723 2565 flags.go:64] FLAG: --reserved-memory="" Jul 07 15:41:00.366447 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.037727 2565 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 07 15:41:00.366447 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.037729 2565 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 07 15:41:00.366447 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.037732 2565 flags.go:64] FLAG: --rotate-certificates="false" Jul 07 15:41:00.366447 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.037735 2565 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 07 15:41:00.366447 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.037738 2565 flags.go:64] FLAG: --runonce="false" Jul 07 15:41:00.366447 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.037741 2565 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 07 15:41:00.366447 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.037744 2565 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 07 15:41:00.366447 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.037747 2565 flags.go:64] FLAG: --seccomp-default="false" Jul 07 15:41:00.366447 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.037750 2565 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 07 15:41:00.366447 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.037753 2565 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 07 15:41:00.370748 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.037756 2565 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 07 15:41:00.370748 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.037759 2565 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 07 15:41:00.370748 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.037761 2565 flags.go:64] FLAG: --storage-driver-password="root" Jul 07 15:41:00.370748 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.037764 2565 flags.go:64] FLAG: --storage-driver-secure="false" Jul 07 15:41:00.370748 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.037767 2565 flags.go:64] FLAG: --storage-driver-table="stats" Jul 07 15:41:00.370748 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.037770 2565 flags.go:64] FLAG: --storage-driver-user="root" Jul 07 15:41:00.370748 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.037774 2565 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 07 15:41:00.370748 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.037777 2565 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 07 15:41:00.370748 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.037805 2565 flags.go:64] FLAG: --system-cgroups="" Jul 07 15:41:00.370748 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.037810 2565 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 07 15:41:00.370748 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.037816 2565 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 07 15:41:00.370748 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.037819 2565 flags.go:64] FLAG: --tls-cert-file="" Jul 07 15:41:00.370748 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.037822 2565 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 07 15:41:00.370748 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.037827 2565 flags.go:64] FLAG: --tls-min-version="" Jul 07 15:41:00.370748 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.037830 2565 flags.go:64] FLAG: --tls-private-key-file="" Jul 07 15:41:00.370748 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.037833 2565 flags.go:64] FLAG: --topology-manager-policy="none" Jul 07 15:41:00.370748 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.037837 2565 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 07 15:41:00.370748 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.037842 2565 flags.go:64] FLAG: --topology-manager-scope="container" Jul 07 15:41:00.370748 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.037845 2565 flags.go:64] FLAG: --v="2" Jul 07 15:41:00.370748 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.037849 2565 flags.go:64] FLAG: --version="false" Jul 07 15:41:00.370748 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.037852 2565 flags.go:64] FLAG: --vmodule="" Jul 07 15:41:00.370748 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.037856 2565 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 07 15:41:00.370748 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.037859 2565 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 07 15:41:00.370748 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.037973 2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 07 15:41:00.370748 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.037977 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 07 15:41:00.371553 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.037981 2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 07 15:41:00.371553 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.037984 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 07 15:41:00.371553 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.037987 2565 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 07 15:41:00.371553 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.037990 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 07 15:41:00.371553 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.037992 2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 07 15:41:00.371553 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.037995 2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 07 15:41:00.371553 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.037999 2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 07 15:41:00.371553 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.038002 2565 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 07 15:41:00.371553 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.038005 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 07 15:41:00.371553 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.038008 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 07 15:41:00.371553 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.038011 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 07 15:41:00.371553 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.038013 2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 07 15:41:00.371553 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.038016 2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 07 15:41:00.371553 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.038020 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 07 15:41:00.371553 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.038023 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 07 15:41:00.371553 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.038026 2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 07 15:41:00.371553 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.038029 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 07 15:41:00.371553 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.038031 2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 07 15:41:00.371553 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.038033 2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 07 15:41:00.372102 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.038036 2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 07 15:41:00.372102 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.038038 2565 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 07 15:41:00.372102 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.038041 2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 07 15:41:00.372102 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.038043 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 07 15:41:00.372102 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.038046 2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 07 15:41:00.372102 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.038050 2565 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 07 15:41:00.372102 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.038052 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 07 15:41:00.372102 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.038055 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 07 15:41:00.372102 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.038057 2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 07 15:41:00.372102 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.038060 2565 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 07 15:41:00.372102 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.038062 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 07 15:41:00.372102 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.038065 2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 07 15:41:00.372102 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.038068 2565 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 07 15:41:00.372102 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.038070 2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 07 15:41:00.372102 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.038073 2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 07 15:41:00.372102 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.038075 2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 07 15:41:00.372102 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.038078 2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 07 15:41:00.372102 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.038080 2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 07 15:41:00.372102 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.038083 2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 07 15:41:00.372102 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.038087 2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 07 15:41:00.372692 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.038090 2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 07 15:41:00.372692 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.038094 2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 07 15:41:00.372692 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.038097 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 07 15:41:00.372692 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.038100 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 07 15:41:00.372692 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.038103 2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 07 15:41:00.372692 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.038105 2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 07 15:41:00.372692 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.038110 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 07 15:41:00.372692 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.038112 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 07 15:41:00.372692 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.038115 2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 07 15:41:00.372692 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.038118 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 07 15:41:00.372692 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.038120 2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 07 15:41:00.372692 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.038122 2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 07 15:41:00.372692 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.038125 2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 07 15:41:00.372692 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.038128 2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 07 15:41:00.372692 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.038131 2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 07 15:41:00.372692 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.038133 2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 07 15:41:00.372692 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.038136 2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 07 15:41:00.372692 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.038140 2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 07 15:41:00.372692 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.038143 2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 07 15:41:00.372692 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.038145 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 07 15:41:00.373266 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.038148 2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 07 15:41:00.373266 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.038150 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 07 15:41:00.373266 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.038153 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 07 15:41:00.373266 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.038155 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 07 15:41:00.373266 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.038158 2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 07 15:41:00.373266 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.038161 2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 07 15:41:00.373266 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.038163 2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 07 15:41:00.373266 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.038165 2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 07 15:41:00.373266 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.038168 2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 07 15:41:00.373266 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.038171 2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 07 15:41:00.373266 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.038173 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 07 15:41:00.373266 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.038176 2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 07 15:41:00.373266 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.038178 2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 07 15:41:00.373266 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.038181 2565 feature_gate.go:328] unrecognized feature gate: Example Jul 07 15:41:00.373266 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.038184 2565 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 07 15:41:00.373266 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.038186 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 07 15:41:00.373266 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.038189 2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 07 15:41:00.373266 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.038191 2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 07 15:41:00.373266 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.038195 2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 07 15:41:00.373266 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.038197 2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 07 15:41:00.373859 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.038200 2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 07 15:41:00.373859 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.038203 2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 07 15:41:00.373859 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.038205 2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 07 15:41:00.373859 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.038207 2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 07 15:41:00.373859 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.038210 2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 07 15:41:00.373859 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.038213 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 07 15:41:00.373859 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.038215 2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 07 15:41:00.373859 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.038218 2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 07 15:41:00.373859 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.038220 2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 07 15:41:00.373859 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.038225 2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 07 15:41:00.373859 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.038228 2565 feature_gate.go:328] unrecognized feature gate: Example2 Jul 07 15:41:00.373859 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.038231 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 07 15:41:00.373859 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.038233 2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 07 15:41:00.373859 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.038236 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 07 15:41:00.373859 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.038238 2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 07 15:41:00.373859 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.038241 2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 07 15:41:00.373859 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.038244 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 07 15:41:00.373859 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.038246 2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 07 15:41:00.374386 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.038732 2565 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 07 15:41:00.374386 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.046860 2565 server.go:529] "Kubelet version" kubeletVersion="v1.34.8" Jul 07 15:41:00.374386 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.046877 2565 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 07 15:41:00.374386 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.048178 2565 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 07 15:41:00.374386 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.048189 2565 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 07 15:41:00.374386 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.048247 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 07 15:41:00.374386 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.048252 2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 07 15:41:00.374386 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.048256 2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 07 15:41:00.374386 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.048259 2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 07 15:41:00.374386 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.048262 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 07 15:41:00.374386 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.048265 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 07 15:41:00.374386 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.048268 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 07 15:41:00.374386 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.048271 2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 07 15:41:00.374386 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.048275 2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 07 15:41:00.374386 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.048278 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 07 15:41:00.374876 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.048280 2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 07 15:41:00.374876 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.048283 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 07 15:41:00.374876 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.048286 2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 07 15:41:00.374876 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.048288 2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 07 15:41:00.374876 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.048291 2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 07 15:41:00.374876 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.048293 2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 07 15:41:00.374876 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.048296 2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 07 15:41:00.374876 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.048299 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 07 15:41:00.374876 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.048302 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 07 15:41:00.374876 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.048304 2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 07 15:41:00.374876 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.048307 2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 07 15:41:00.374876 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.048310 2565 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 07 15:41:00.374876 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.048313 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 07 15:41:00.374876 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.048317 2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 07 15:41:00.374876 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.048320 2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 07 15:41:00.374876 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.048322 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 07 15:41:00.374876 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.048325 2565 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 07 15:41:00.374876 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.048328 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 07 15:41:00.374876 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.048331 2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 07 15:41:00.450889 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.048333 2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 07 15:41:00.450889 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.048336 2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 07 15:41:00.450889 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.048338 2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 07 15:41:00.450889 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.048341 2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 07 15:41:00.450889 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.048343 2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 07 15:41:00.450889 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.048346 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 07 15:41:00.450889 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.048350 2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 07 15:41:00.450889 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.048353 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 07 15:41:00.450889 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.048356 2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 07 15:41:00.450889 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.048359 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 07 15:41:00.450889 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.048362 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 07 15:41:00.450889 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.048365 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 07 15:41:00.450889 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.048368 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 07 15:41:00.450889 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.048371 2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 07 15:41:00.450889 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.048374 2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 07 15:41:00.450889 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.048376 2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 07 15:41:00.450889 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.048379 2565 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 07 15:41:00.450889 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.048383 2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 07 15:41:00.450889 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.048385 2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 07 15:41:00.450889 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.048388 2565 feature_gate.go:328] unrecognized feature gate: Example2 Jul 07 15:41:00.451655 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.048391 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 07 15:41:00.451655 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.048393 2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 07 15:41:00.451655 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.048396 2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 07 15:41:00.451655 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.048398 2565 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 07 15:41:00.451655 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.048401 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 07 15:41:00.451655 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.048404 2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 07 15:41:00.451655 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.048409 2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 07 15:41:00.451655 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.048412 2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 07 15:41:00.451655 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.048416 2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 07 15:41:00.451655 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.048419 2565 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 07 15:41:00.451655 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.048422 2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 07 15:41:00.451655 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.048425 2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 07 15:41:00.451655 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.048427 2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 07 15:41:00.451655 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.048430 2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 07 15:41:00.451655 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.048433 2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 07 15:41:00.451655 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.048435 2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 07 15:41:00.451655 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.048438 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 07 15:41:00.451655 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.048440 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 07 15:41:00.451655 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.048443 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 07 15:41:00.451655 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.048445 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 07 15:41:00.452295 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.048448 2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 07 15:41:00.452295 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.048453 2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 07 15:41:00.452295 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.048456 2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 07 15:41:00.452295 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.048459 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 07 15:41:00.452295 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.048461 2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 07 15:41:00.452295 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.048464 2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 07 15:41:00.452295 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.048467 2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 07 15:41:00.452295 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.048469 2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 07 15:41:00.452295 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.048472 2565 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 07 15:41:00.452295 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.048475 2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 07 15:41:00.452295 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.048478 2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 07 15:41:00.452295 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.048481 2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 07 15:41:00.452295 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.048483 2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 07 15:41:00.452295 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.048486 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 07 15:41:00.452295 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.048489 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 07 15:41:00.452295 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.048491 2565 feature_gate.go:328] unrecognized feature gate: Example Jul 07 15:41:00.452295 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.048494 2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 07 15:41:00.452295 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.048497 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 07 15:41:00.452295 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.048501 2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 07 15:41:00.452295 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.048504 2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 07 15:41:00.452941 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.048509 2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 07 15:41:00.452941 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.048511 2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 07 15:41:00.452941 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.048514 2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 07 15:41:00.452941 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.048517 2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 07 15:41:00.452941 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.048519 2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 07 15:41:00.452941 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.048523 2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 07 15:41:00.452941 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.048527 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 07 15:41:00.452941 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.048530 2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 07 15:41:00.452941 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.048533 2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 07 15:41:00.452941 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.048536 2565 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 07 15:41:00.452941 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.048540 2565 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 07 15:41:00.452941 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.048676 2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 07 15:41:00.452941 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.048681 2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 07 15:41:00.452941 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.048684 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 07 15:41:00.453361 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.048687 2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 07 15:41:00.453361 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.048691 2565 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 07 15:41:00.453361 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.048694 2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 07 15:41:00.453361 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.048697 2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 07 15:41:00.453361 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.048699 2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 07 15:41:00.453361 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.048702 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 07 15:41:00.453361 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.048705 2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 07 15:41:00.453361 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.048708 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 07 15:41:00.453361 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.048710 2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 07 15:41:00.453361 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.048713 2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 07 15:41:00.453361 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.048716 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 07 15:41:00.453361 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.048719 2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 07 15:41:00.453361 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.048721 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 07 15:41:00.453361 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.048724 2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 07 15:41:00.453361 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.048727 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 07 15:41:00.453361 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.048729 2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 07 15:41:00.453361 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.048732 2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 07 15:41:00.453361 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.048736 2565 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 07 15:41:00.453361 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.048742 2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 07 15:41:00.453361 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.048745 2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 07 15:41:00.454012 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.048748 2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 07 15:41:00.454012 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.048751 2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 07 15:41:00.454012 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.048754 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 07 15:41:00.454012 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.048757 2565 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 07 15:41:00.454012 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.048760 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 07 15:41:00.454012 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.048763 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 07 15:41:00.454012 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.048766 2565 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 07 15:41:00.454012 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.048768 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 07 15:41:00.454012 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.048771 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 07 15:41:00.454012 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.048774 2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 07 15:41:00.454012 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.048776 2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 07 15:41:00.454012 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.048795 2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 07 15:41:00.454012 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.048798 2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 07 15:41:00.454012 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.048801 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 07 15:41:00.454012 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.048803 2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 07 15:41:00.454012 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.048806 2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 07 15:41:00.454012 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.048809 2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 07 15:41:00.454012 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.048811 2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 07 15:41:00.454012 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.048814 2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 07 15:41:00.454012 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.048816 2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 07 15:41:00.454661 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.048819 2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 07 15:41:00.454661 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.048822 2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 07 15:41:00.454661 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.048825 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 07 15:41:00.454661 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.048827 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 07 15:41:00.454661 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.048830 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 07 15:41:00.454661 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.048833 2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 07 15:41:00.454661 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.048835 2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 07 15:41:00.454661 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.048838 2565 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 07 15:41:00.454661 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.048841 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 07 15:41:00.454661 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.048845 2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 07 15:41:00.454661 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.048849 2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 07 15:41:00.454661 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.048851 2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 07 15:41:00.454661 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.048854 2565 feature_gate.go:328] unrecognized feature gate: Example Jul 07 15:41:00.454661 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.048857 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 07 15:41:00.454661 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.048859 2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 07 15:41:00.454661 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.048862 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 07 15:41:00.454661 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.048865 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 07 15:41:00.454661 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.048867 2565 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 07 15:41:00.454661 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.048870 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 07 15:41:00.454661 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.048872 2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 07 15:41:00.455340 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.048875 2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 07 15:41:00.455340 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.048877 2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 07 15:41:00.455340 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.048880 2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 07 15:41:00.455340 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.048882 2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 07 15:41:00.455340 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.048886 2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 07 15:41:00.455340 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.048890 2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 07 15:41:00.455340 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.048892 2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 07 15:41:00.455340 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.048895 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 07 15:41:00.455340 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.048897 2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 07 15:41:00.455340 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.048900 2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 07 15:41:00.455340 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.048902 2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 07 15:41:00.455340 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.048905 2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 07 15:41:00.455340 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.048907 2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 07 15:41:00.455340 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.048910 2565 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 07 15:41:00.455340 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.048912 2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 07 15:41:00.455340 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.048915 2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 07 15:41:00.455340 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.048918 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 07 15:41:00.455340 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.048921 2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 07 15:41:00.455340 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.048923 2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 07 15:41:00.455999 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.048925 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 07 15:41:00.455999 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.048929 2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 07 15:41:00.455999 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.048934 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 07 15:41:00.455999 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.048938 2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 07 15:41:00.455999 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.048941 2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 07 15:41:00.455999 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.048944 2565 feature_gate.go:328] unrecognized feature gate: Example2 Jul 07 15:41:00.455999 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.048947 2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 07 15:41:00.455999 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.048950 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 07 15:41:00.455999 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.048952 2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 07 15:41:00.455999 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.048955 2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 07 15:41:00.455999 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.048957 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 07 15:41:00.455999 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.048960 2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 07 15:41:00.455999 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.048962 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 07 15:41:00.455999 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.048965 2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 07 15:41:00.455999 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.048968 2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 07 15:41:00.455999 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.048970 2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 07 15:41:00.455999 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:00.048972 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 07 15:41:00.456659 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.048977 2565 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 07 15:41:00.456659 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.049114 2565 server.go:962] "Client rotation is on, will bootstrap in background" Jul 07 15:41:00.456659 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.053740 2565 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 07 15:41:00.456659 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.055026 2565 server.go:1019] "Starting client certificate rotation" Jul 07 15:41:00.456659 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.055125 2565 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 07 15:41:00.456659 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.055159 2565 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 07 15:41:00.456659 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.057333 2565 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 07 15:41:00.456659 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.059176 2565 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 07 15:41:00.456659 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.069593 2565 log.go:25] "Validated CRI v1 runtime API" Jul 07 15:41:00.456659 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.073989 2565 log.go:25] "Validated CRI v1 image API" Jul 07 15:41:00.456659 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.074763 2565 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 07 15:41:00.456659 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.079398 2565 fs.go:135] Filesystem UUIDs: map[29ad1a17-1e9a-487e-a69a-2044578f428b:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 8e428995-cf8e-4111-9980-b59460de01bb:/dev/nvme0n1p3] Jul 07 15:41:00.457099 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.079419 2565 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 07 15:41:00.457325 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.085550 2565 manager.go:216] Machine: {Timestamp:2026-07-07 15:41:00.084356872 +0000 UTC m=+0.378643757 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099766 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec25bbef8f80ba684283709ca8779fef SystemUUID:ec25bbef-8f80-ba68-4283-709ca8779fef BootID:ac6bb38b-688e-4efb-acc6-61bbae91724c Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:f1:17:dd:d9:05 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:f1:17:dd:d9:05 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:36:11:98:7b:4a:1a Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 07 15:41:00.457325 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.085651 2565 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 07 15:41:00.457325 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.085735 2565 manager.go:232] Version: {KernelVersion:5.14.0-570.125.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260630-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 07 15:41:00.457325 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.088717 2565 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 07 15:41:00.457325 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.088744 2565 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-132-65.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 07 15:41:00.457325 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.090164 2565 topology_manager.go:138] "Creating topology manager with none policy" Jul 07 15:41:00.457325 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.090173 2565 container_manager_linux.go:309] "Creating device plugin manager" Jul 07 15:41:00.457325 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.090182 2565 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 07 15:41:00.460344 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.090746 2565 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 07 15:41:00.460344 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.092044 2565 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 07 15:41:00.460344 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.092303 2565 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 07 15:41:00.460344 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.094719 2565 state_mem.go:36] "Initialized new in-memory state store" Jul 07 15:41:00.460344 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.094857 2565 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 07 15:41:00.460344 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.094886 2565 kubelet.go:486] "Attempting to sync node with API server" Jul 07 15:41:00.460344 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.094907 2565 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 07 15:41:00.460344 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.094931 2565 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 07 15:41:00.460344 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.094943 2565 kubelet.go:398] "Adding apiserver pod source" Jul 07 15:41:00.460344 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.094955 2565 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 07 15:41:00.460344 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.095382 2565 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 07 15:41:00.460344 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.095437 2565 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 07 15:41:00.460344 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.098830 2565 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.8-3.rhaos4.21.git91b4b0b.el9" apiVersion="v1" Jul 07 15:41:00.460344 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.104381 2565 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 07 15:41:00.460344 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.104409 2565 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 07 15:41:00.460344 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.105300 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 07 15:41:00.460344 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.105315 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 07 15:41:00.460344 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.105321 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 07 15:41:00.460344 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.105326 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 07 15:41:00.460344 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.105335 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 07 15:41:00.460344 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.105341 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 07 15:41:00.460344 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.105349 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 07 15:41:00.460344 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.105354 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 07 15:41:00.460344 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.105361 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 07 15:41:00.460344 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.105367 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 07 15:41:00.460344 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.105375 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 07 15:41:00.460344 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.105381 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 07 15:41:00.460344 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.106355 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 07 15:41:00.460344 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.106362 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 07 15:41:00.460344 ip-10-0-132-65 kubenswrapper[2565]: E0707 15:41:00.108551 2565 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-132-65.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 07 15:41:00.460344 ip-10-0-132-65 kubenswrapper[2565]: E0707 15:41:00.108579 2565 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 07 15:41:00.460344 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.109606 2565 server.go:1268] "Started kubelet" Jul 07 15:41:00.460344 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.109701 2565 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 07 15:41:00.460344 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.109769 2565 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 07 15:41:00.460344 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.109760 2565 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 07 15:41:00.460344 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.111380 2565 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 07 15:41:00.460344 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.112172 2565 server.go:310] "Adding debug handlers to kubelet server" Jul 07 15:41:00.460344 ip-10-0-132-65 kubenswrapper[2565]: E0707 15:41:00.114237 2565 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 07 15:41:00.460344 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.118431 2565 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 07 15:41:00.460344 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.118437 2565 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 07 15:41:00.461619 ip-10-0-132-65 kubenswrapper[2565]: E0707 15:41:00.118513 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-65.ec2.internal\" not found" Jul 07 15:41:00.461619 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.118988 2565 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 07 15:41:00.461619 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.118996 2565 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 07 15:41:00.461619 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.119008 2565 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 07 15:41:00.461619 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.119155 2565 reconstruct.go:97] "Volume reconstruction finished" Jul 07 15:41:00.461619 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.119165 2565 reconciler.go:29] "Reconciler: start to sync state" Jul 07 15:41:00.461619 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.120042 2565 factory.go:55] Registering systemd factory Jul 07 15:41:00.461619 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.120058 2565 factory.go:223] Registration of the systemd container factory successfully Jul 07 15:41:00.461619 ip-10-0-132-65 kubenswrapper[2565]: E0707 15:41:00.119072 2565 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-65.ec2.internal.18c00c31b77588bb default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-65.ec2.internal,UID:ip-10-0-132-65.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-132-65.ec2.internal,},FirstTimestamp:2026-07-07 15:41:00.109580475 +0000 UTC m=+0.403867358,LastTimestamp:2026-07-07 15:41:00.109580475 +0000 UTC m=+0.403867358,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-65.ec2.internal,}" Jul 07 15:41:00.461619 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.120440 2565 factory.go:153] Registering CRI-O factory Jul 07 15:41:00.461619 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.120455 2565 factory.go:223] Registration of the crio container factory successfully Jul 07 15:41:00.461619 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.120518 2565 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 07 15:41:00.461619 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.120541 2565 factory.go:103] Registering Raw factory Jul 07 15:41:00.461619 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.120557 2565 manager.go:1196] Started watching for new ooms in manager Jul 07 15:41:00.461619 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.121031 2565 manager.go:349] Starting recovery of all containers Jul 07 15:41:00.461619 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.130035 2565 manager.go:354] Recovery completed Jul 07 15:41:00.461619 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.134231 2565 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-65.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 07 15:41:00.461619 ip-10-0-132-65 kubenswrapper[2565]: E0707 15:41:00.134255 2565 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 07 15:41:00.461619 ip-10-0-132-65 kubenswrapper[2565]: E0707 15:41:00.134534 2565 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-132-65.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 07 15:41:00.461619 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.134765 2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 07 15:41:00.461619 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.137403 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-65.ec2.internal" event="NodeHasSufficientMemory" Jul 07 15:41:00.461619 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.137429 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-65.ec2.internal" event="NodeHasNoDiskPressure" Jul 07 15:41:00.461619 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.137439 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-65.ec2.internal" event="NodeHasSufficientPID" Jul 07 15:41:00.461619 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.138032 2565 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 07 15:41:00.461619 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.138044 2565 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 07 15:41:00.461619 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.138033 2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 07 15:41:00.461619 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.138064 2565 state_mem.go:36] "Initialized new in-memory state store" Jul 07 15:41:00.461619 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.138656 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-65.ec2.internal" event="NodeHasSufficientMemory" Jul 07 15:41:00.461619 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.138685 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-65.ec2.internal" event="NodeHasNoDiskPressure" Jul 07 15:41:00.461619 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.138696 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-65.ec2.internal" event="NodeHasSufficientPID" Jul 07 15:41:00.461619 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.139344 2565 policy_none.go:49] "None policy: Start" Jul 07 15:41:00.461619 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.139356 2565 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 07 15:41:00.468448 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.139366 2565 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 07 15:41:00.468448 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.140383 2565 policy_none.go:47] "Start" Jul 07 15:41:00.468448 ip-10-0-132-65 kubenswrapper[2565]: E0707 15:41:00.148308 2565 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-65.ec2.internal.18c00c31b91e42cb default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-65.ec2.internal,UID:ip-10-0-132-65.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-132-65.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-132-65.ec2.internal,},FirstTimestamp:2026-07-07 15:41:00.137415371 +0000 UTC m=+0.431702254,LastTimestamp:2026-07-07 15:41:00.137415371 +0000 UTC m=+0.431702254,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-65.ec2.internal,}" Jul 07 15:41:00.468448 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.150333 2565 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-j24db" Jul 07 15:41:00.468448 ip-10-0-132-65 kubenswrapper[2565]: E0707 15:41:00.151621 2565 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-65.ec2.internal.18c00c31b91e898a default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-65.ec2.internal,UID:ip-10-0-132-65.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-132-65.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-132-65.ec2.internal,},FirstTimestamp:2026-07-07 15:41:00.137433482 +0000 UTC m=+0.431720366,LastTimestamp:2026-07-07 15:41:00.137433482 +0000 UTC m=+0.431720366,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-65.ec2.internal,}" Jul 07 15:41:00.468448 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.161564 2565 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-j24db" Jul 07 15:41:00.468448 ip-10-0-132-65 kubenswrapper[2565]: E0707 15:41:00.168261 2565 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-65.ec2.internal.18c00c31b91eae5c default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-65.ec2.internal,UID:ip-10-0-132-65.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-132-65.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-132-65.ec2.internal,},FirstTimestamp:2026-07-07 15:41:00.137442908 +0000 UTC m=+0.431729791,LastTimestamp:2026-07-07 15:41:00.137442908 +0000 UTC m=+0.431729791,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-65.ec2.internal,}" Jul 07 15:41:00.468448 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.202340 2565 manager.go:337] "Starting Device Plugin manager" Jul 07 15:41:00.468448 ip-10-0-132-65 kubenswrapper[2565]: E0707 15:41:00.202369 2565 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 07 15:41:00.468448 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.202379 2565 server.go:87] "Starting device plugin registration server" Jul 07 15:41:00.468448 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.202632 2565 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 07 15:41:00.468448 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.202649 2565 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 07 15:41:00.468448 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.202808 2565 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 07 15:41:00.468448 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.202882 2565 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 07 15:41:00.468448 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.202890 2565 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 07 15:41:00.468448 ip-10-0-132-65 kubenswrapper[2565]: E0707 15:41:00.203333 2565 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 07 15:41:00.468448 ip-10-0-132-65 kubenswrapper[2565]: E0707 15:41:00.203388 2565 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-132-65.ec2.internal\" not found" Jul 07 15:41:00.468448 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.303177 2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 07 15:41:00.468448 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.304262 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-65.ec2.internal" event="NodeHasSufficientMemory" Jul 07 15:41:00.469123 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.304292 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-65.ec2.internal" event="NodeHasNoDiskPressure" Jul 07 15:41:00.469123 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.304302 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-65.ec2.internal" event="NodeHasSufficientPID" Jul 07 15:41:00.469123 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.304332 2565 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-132-65.ec2.internal" Jul 07 15:41:00.469123 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.315227 2565 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 07 15:41:00.469123 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.316352 2565 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 07 15:41:00.469123 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.316384 2565 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 07 15:41:00.469123 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.316403 2565 kubelet.go:2448] "Starting kubelet main sync loop" Jul 07 15:41:00.469123 ip-10-0-132-65 kubenswrapper[2565]: E0707 15:41:00.316436 2565 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 07 15:41:00.469123 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.317747 2565 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-132-65.ec2.internal" Jul 07 15:41:00.469123 ip-10-0-132-65 kubenswrapper[2565]: E0707 15:41:00.317772 2565 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-132-65.ec2.internal\": node \"ip-10-0-132-65.ec2.internal\" not found" Jul 07 15:41:00.469123 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.321229 2565 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 07 15:41:00.469123 ip-10-0-132-65 kubenswrapper[2565]: E0707 15:41:00.336175 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-65.ec2.internal\" not found" Jul 07 15:41:00.469123 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.416506 2565 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-132-65.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-65.ec2.internal"] Jul 07 15:41:00.469123 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.416723 2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 07 15:41:00.469123 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.418263 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-65.ec2.internal" event="NodeHasSufficientMemory" Jul 07 15:41:00.469123 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.418294 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-65.ec2.internal" event="NodeHasNoDiskPressure" Jul 07 15:41:00.469123 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.418304 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-65.ec2.internal" event="NodeHasSufficientPID" Jul 07 15:41:00.469123 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.419934 2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 07 15:41:00.469123 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.419967 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9fc4f78a01c7da970c3b1d6baab931c6-config\") pod \"kube-apiserver-proxy-ip-10-0-132-65.ec2.internal\" (UID: \"9fc4f78a01c7da970c3b1d6baab931c6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-65.ec2.internal" Jul 07 15:41:00.469123 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.420052 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-65.ec2.internal" Jul 07 15:41:00.469123 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.420093 2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 07 15:41:00.469123 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.420677 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-65.ec2.internal" event="NodeHasSufficientMemory" Jul 07 15:41:00.469123 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.420702 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-65.ec2.internal" event="NodeHasNoDiskPressure" Jul 07 15:41:00.469123 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.420715 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-65.ec2.internal" event="NodeHasSufficientPID" Jul 07 15:41:00.469123 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.421037 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-65.ec2.internal" event="NodeHasSufficientMemory" Jul 07 15:41:00.469123 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.421057 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-65.ec2.internal" event="NodeHasNoDiskPressure" Jul 07 15:41:00.469123 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.421066 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-65.ec2.internal" event="NodeHasSufficientPID" Jul 07 15:41:00.469123 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.421904 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-65.ec2.internal" Jul 07 15:41:00.469123 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.421925 2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 07 15:41:00.469123 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.422521 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-65.ec2.internal" event="NodeHasSufficientMemory" Jul 07 15:41:00.469123 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.422538 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-65.ec2.internal" event="NodeHasNoDiskPressure" Jul 07 15:41:00.469123 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.422548 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-65.ec2.internal" event="NodeHasSufficientPID" Jul 07 15:41:00.469123 ip-10-0-132-65 kubenswrapper[2565]: E0707 15:41:00.436228 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-65.ec2.internal\" not found" Jul 07 15:41:00.590309 ip-10-0-132-65 kubenswrapper[2565]: E0707 15:41:00.581831 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-65.ec2.internal\" not found" Jul 07 15:41:00.590309 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.581901 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9fc4f78a01c7da970c3b1d6baab931c6-config\") pod \"kube-apiserver-proxy-ip-10-0-132-65.ec2.internal\" (UID: \"9fc4f78a01c7da970c3b1d6baab931c6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-65.ec2.internal" Jul 07 15:41:00.590309 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.581935 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9fc4f78a01c7da970c3b1d6baab931c6-config\") pod \"kube-apiserver-proxy-ip-10-0-132-65.ec2.internal\" (UID: \"9fc4f78a01c7da970c3b1d6baab931c6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-65.ec2.internal" Jul 07 15:41:00.688558 ip-10-0-132-65 kubenswrapper[2565]: E0707 15:41:00.599601 2565 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-65.ec2.internal\" not found" node="ip-10-0-132-65.ec2.internal" Jul 07 15:41:00.688558 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.601863 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-65.ec2.internal" Jul 07 15:41:00.688558 ip-10-0-132-65 kubenswrapper[2565]: E0707 15:41:00.604412 2565 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-65.ec2.internal\" not found" node="ip-10-0-132-65.ec2.internal" Jul 07 15:41:00.688558 ip-10-0-132-65 kubenswrapper[2565]: E0707 15:41:00.681890 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-65.ec2.internal\" not found" Jul 07 15:41:00.688558 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.681984 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/98ae1d05f22dc745f3b265fc9d76c5f3-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-65.ec2.internal\" (UID: \"98ae1d05f22dc745f3b265fc9d76c5f3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-65.ec2.internal" Jul 07 15:41:00.688558 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.682008 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/98ae1d05f22dc745f3b265fc9d76c5f3-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-65.ec2.internal\" (UID: \"98ae1d05f22dc745f3b265fc9d76c5f3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-65.ec2.internal" Jul 07 15:41:02.189139 ip-10-0-132-65 kubenswrapper[2565]: E0707 15:41:00.782759 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-65.ec2.internal\" not found" Jul 07 15:41:02.189139 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.782772 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/98ae1d05f22dc745f3b265fc9d76c5f3-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-65.ec2.internal\" (UID: \"98ae1d05f22dc745f3b265fc9d76c5f3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-65.ec2.internal" Jul 07 15:41:02.189139 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.782823 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/98ae1d05f22dc745f3b265fc9d76c5f3-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-65.ec2.internal\" (UID: \"98ae1d05f22dc745f3b265fc9d76c5f3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-65.ec2.internal" Jul 07 15:41:02.189139 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.782869 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/98ae1d05f22dc745f3b265fc9d76c5f3-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-65.ec2.internal\" (UID: \"98ae1d05f22dc745f3b265fc9d76c5f3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-65.ec2.internal" Jul 07 15:41:02.189139 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:00.782907 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/98ae1d05f22dc745f3b265fc9d76c5f3-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-65.ec2.internal\" (UID: \"98ae1d05f22dc745f3b265fc9d76c5f3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-65.ec2.internal" Jul 07 15:41:02.276196 ip-10-0-132-65 kubenswrapper[2565]: E0707 15:41:02.220923 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-65.ec2.internal\" not found" Jul 07 15:41:02.276196 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.245078 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-65.ec2.internal" Jul 07 15:41:02.359068 ip-10-0-132-65 kubenswrapper[2565]: E0707 15:41:02.321922 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-65.ec2.internal\" not found" Jul 07 15:41:02.359068 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.356453 2565 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 07 15:41:02.456420 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.392611 2565 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 07 15:41:02.456420 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.392766 2565 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 07 15:41:02.456420 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.392789 2565 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 07 15:41:02.456420 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.432984 2565 apiserver.go:52] "Watching apiserver" Jul 07 15:41:02.457446 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.457053 2565 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 07 15:41:02.457446 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.457079 2565 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-65.ec2.internal" Jul 07 15:41:02.543472 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.465032 2565 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 07 15:41:02.543472 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.465335 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-jzprp","openshift-multus/network-metrics-daemon-sdzxq","openshift-network-diagnostics/network-check-target-94vlx","openshift-network-operator/iptables-alerter-lrpv9","kube-system/konnectivity-agent-m4wtp","openshift-cluster-node-tuning-operator/tuned-4glxt","openshift-image-registry/node-ca-9fmf8","openshift-ovn-kubernetes/ovnkube-node-942xh","kube-system/global-pull-secret-syncer-6qzkn","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8x25","openshift-dns/node-resolver-qggz6","openshift-multus/multus-additional-cni-plugins-f7x65"] Jul 07 15:41:02.543472 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.467335 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-jzprp" Jul 07 15:41:02.543472 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.469218 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sdzxq" Jul 07 15:41:02.543472 ip-10-0-132-65 kubenswrapper[2565]: E0707 15:41:02.469279 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sdzxq" podUID="301c1027-169c-47f5-a3ea-89cde78790d2" Jul 07 15:41:02.543472 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.470232 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-94vlx" Jul 07 15:41:02.543472 ip-10-0-132-65 kubenswrapper[2565]: E0707 15:41:02.470277 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-94vlx" podUID="e1f2bb91-99ac-4353-b790-622cb6d40663" Jul 07 15:41:02.543472 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.471888 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-lrpv9" Jul 07 15:41:02.543472 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.472300 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-m4wtp" Jul 07 15:41:02.543472 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.481354 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-f7x65" Jul 07 15:41:02.543472 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.481599 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-4glxt" Jul 07 15:41:02.543472 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.481843 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-9fmf8" Jul 07 15:41:02.543472 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.482125 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-942xh" Jul 07 15:41:02.543472 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.482418 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6qzkn" Jul 07 15:41:02.543472 ip-10-0-132-65 kubenswrapper[2565]: E0707 15:41:02.482465 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6qzkn" podUID="0c5071d6-cead-4992-a7e9-9a752b955cae" Jul 07 15:41:02.543472 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.482583 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8x25" Jul 07 15:41:02.543472 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.482784 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-qggz6" Jul 07 15:41:02.543472 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.491548 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 07 15:41:02.543472 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.491560 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 07 15:41:02.543472 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.491591 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 07 15:41:02.543472 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.491549 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 07 15:41:02.543472 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.491659 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 07 15:41:02.543472 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.492727 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 07 15:41:02.543472 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.492968 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 07 15:41:02.543472 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.493120 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-n9wxm\"" Jul 07 15:41:02.544482 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.493144 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 07 15:41:02.544482 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.493262 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 07 15:41:02.544482 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.493306 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-fsrlh\"" Jul 07 15:41:02.544482 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.493460 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-c9ppb\"" Jul 07 15:41:02.544482 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.494066 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 07 15:41:02.544482 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.494078 2565 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 07 15:41:02.544482 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.494080 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 07 15:41:02.544482 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.494117 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 07 15:41:02.544482 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.494156 2565 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-65.ec2.internal" Jul 07 15:41:02.544482 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.494508 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-132-65.ec2.internal"] Jul 07 15:41:02.544482 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.494827 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 07 15:41:02.544482 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.494855 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 07 15:41:02.544482 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.495966 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 07 15:41:02.544482 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.495982 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 07 15:41:02.544482 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.495992 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-gl5qw\"" Jul 07 15:41:02.544482 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.496428 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 07 15:41:02.544482 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.496440 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 07 15:41:02.544482 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.496480 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-kcq7p\"" Jul 07 15:41:02.544482 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.496733 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 07 15:41:02.544482 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.496787 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 07 15:41:02.544482 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.496800 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-ngnsj\"" Jul 07 15:41:02.544482 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.496789 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 07 15:41:02.544482 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.497919 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 07 15:41:02.544482 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.498003 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 07 15:41:02.544482 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.498031 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 07 15:41:02.544482 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.498088 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-w2mgr\"" Jul 07 15:41:02.544482 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.498355 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 07 15:41:02.544482 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.498811 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-v9phb\"" Jul 07 15:41:02.544482 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.504846 2565 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-06 15:36:01 +0000 UTC" deadline="2027-12-18 08:14:50.626403219 +0000 UTC" Jul 07 15:41:02.544482 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.504869 2565 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12688h33m48.121536526s" Jul 07 15:41:02.545940 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.508085 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 07 15:41:02.545940 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.508415 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 07 15:41:02.545940 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.508416 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-g9mmh\"" Jul 07 15:41:02.545940 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.529879 2565 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 07 15:41:02.545940 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.529931 2565 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 07 15:41:02.545940 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.530072 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-65.ec2.internal"] Jul 07 15:41:02.636159 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.556763 2565 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-sr45q" Jul 07 15:41:02.636159 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.557506 2565 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 07 15:41:02.636159 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.575585 2565 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-sr45q" Jul 07 15:41:02.636159 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.624482 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4e14c00d-b605-4eee-aaa1-8b07158de8f0-tmp\") pod \"tuned-4glxt\" (UID: \"4e14c00d-b605-4eee-aaa1-8b07158de8f0\") " pod="openshift-cluster-node-tuning-operator/tuned-4glxt" Jul 07 15:41:02.636159 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.624530 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e5c3e573-02fc-4e63-b475-65ee8b00f34b-serviceca\") pod \"node-ca-9fmf8\" (UID: \"e5c3e573-02fc-4e63-b475-65ee8b00f34b\") " pod="openshift-image-registry/node-ca-9fmf8" Jul 07 15:41:02.636159 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.624549 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/31ff16f0-6054-4eba-942a-b07a5a7afac7-host-kubelet\") pod \"ovnkube-node-942xh\" (UID: \"31ff16f0-6054-4eba-942a-b07a5a7afac7\") " pod="openshift-ovn-kubernetes/ovnkube-node-942xh" Jul 07 15:41:02.636159 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.624564 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/31ff16f0-6054-4eba-942a-b07a5a7afac7-run-systemd\") pod \"ovnkube-node-942xh\" (UID: \"31ff16f0-6054-4eba-942a-b07a5a7afac7\") " pod="openshift-ovn-kubernetes/ovnkube-node-942xh" Jul 07 15:41:02.636159 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.624582 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/31ff16f0-6054-4eba-942a-b07a5a7afac7-ovnkube-config\") pod \"ovnkube-node-942xh\" (UID: \"31ff16f0-6054-4eba-942a-b07a5a7afac7\") " pod="openshift-ovn-kubernetes/ovnkube-node-942xh" Jul 07 15:41:02.636159 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.624620 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/57e72845-ffcd-4aa2-91c8-05d2f00b6b49-host-var-lib-kubelet\") pod \"multus-jzprp\" (UID: \"57e72845-ffcd-4aa2-91c8-05d2f00b6b49\") " pod="openshift-multus/multus-jzprp" Jul 07 15:41:02.636159 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.624650 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e6634bed-1f16-4667-ae9e-c08984716ce5-agent-certs\") pod \"konnectivity-agent-m4wtp\" (UID: \"e6634bed-1f16-4667-ae9e-c08984716ce5\") " pod="kube-system/konnectivity-agent-m4wtp" Jul 07 15:41:02.636159 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.624667 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/4e14c00d-b605-4eee-aaa1-8b07158de8f0-etc-tuned\") pod \"tuned-4glxt\" (UID: \"4e14c00d-b605-4eee-aaa1-8b07158de8f0\") " pod="openshift-cluster-node-tuning-operator/tuned-4glxt" Jul 07 15:41:02.636159 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.624682 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/31ff16f0-6054-4eba-942a-b07a5a7afac7-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-942xh\" (UID: \"31ff16f0-6054-4eba-942a-b07a5a7afac7\") " pod="openshift-ovn-kubernetes/ovnkube-node-942xh" Jul 07 15:41:02.636159 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.624698 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m25h7\" (UniqueName: \"kubernetes.io/projected/57e72845-ffcd-4aa2-91c8-05d2f00b6b49-kube-api-access-m25h7\") pod \"multus-jzprp\" (UID: \"57e72845-ffcd-4aa2-91c8-05d2f00b6b49\") " pod="openshift-multus/multus-jzprp" Jul 07 15:41:02.636159 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.624720 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ec51f9ec-f9f7-4390-9954-eb4ab30b8c4d-iptables-alerter-script\") pod \"iptables-alerter-lrpv9\" (UID: \"ec51f9ec-f9f7-4390-9954-eb4ab30b8c4d\") " pod="openshift-network-operator/iptables-alerter-lrpv9" Jul 07 15:41:02.636159 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.624735 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjxct\" (UniqueName: \"kubernetes.io/projected/ec51f9ec-f9f7-4390-9954-eb4ab30b8c4d-kube-api-access-zjxct\") pod \"iptables-alerter-lrpv9\" (UID: \"ec51f9ec-f9f7-4390-9954-eb4ab30b8c4d\") " pod="openshift-network-operator/iptables-alerter-lrpv9" Jul 07 15:41:02.636970 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.624755 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/301c1027-169c-47f5-a3ea-89cde78790d2-metrics-certs\") pod \"network-metrics-daemon-sdzxq\" (UID: \"301c1027-169c-47f5-a3ea-89cde78790d2\") " pod="openshift-multus/network-metrics-daemon-sdzxq" Jul 07 15:41:02.636970 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.624774 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4e14c00d-b605-4eee-aaa1-8b07158de8f0-lib-modules\") pod \"tuned-4glxt\" (UID: \"4e14c00d-b605-4eee-aaa1-8b07158de8f0\") " pod="openshift-cluster-node-tuning-operator/tuned-4glxt" Jul 07 15:41:02.636970 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.624790 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/31ff16f0-6054-4eba-942a-b07a5a7afac7-ovnkube-script-lib\") pod \"ovnkube-node-942xh\" (UID: \"31ff16f0-6054-4eba-942a-b07a5a7afac7\") " pod="openshift-ovn-kubernetes/ovnkube-node-942xh" Jul 07 15:41:02.636970 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.624807 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzqjv\" (UniqueName: \"kubernetes.io/projected/31ff16f0-6054-4eba-942a-b07a5a7afac7-kube-api-access-lzqjv\") pod \"ovnkube-node-942xh\" (UID: \"31ff16f0-6054-4eba-942a-b07a5a7afac7\") " pod="openshift-ovn-kubernetes/ovnkube-node-942xh" Jul 07 15:41:02.636970 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.624825 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0c5071d6-cead-4992-a7e9-9a752b955cae-dbus\") pod \"global-pull-secret-syncer-6qzkn\" (UID: \"0c5071d6-cead-4992-a7e9-9a752b955cae\") " pod="kube-system/global-pull-secret-syncer-6qzkn" Jul 07 15:41:02.636970 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.624862 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e6634bed-1f16-4667-ae9e-c08984716ce5-konnectivity-ca\") pod \"konnectivity-agent-m4wtp\" (UID: \"e6634bed-1f16-4667-ae9e-c08984716ce5\") " pod="kube-system/konnectivity-agent-m4wtp" Jul 07 15:41:02.636970 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.624917 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/76d2d5c2-57b5-4c2d-81f9-73f49cde5fed-registration-dir\") pod \"aws-ebs-csi-driver-node-s8x25\" (UID: \"76d2d5c2-57b5-4c2d-81f9-73f49cde5fed\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8x25" Jul 07 15:41:02.636970 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.624945 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e5c3e573-02fc-4e63-b475-65ee8b00f34b-host\") pod \"node-ca-9fmf8\" (UID: \"e5c3e573-02fc-4e63-b475-65ee8b00f34b\") " pod="openshift-image-registry/node-ca-9fmf8" Jul 07 15:41:02.636970 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.624967 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/31ff16f0-6054-4eba-942a-b07a5a7afac7-host-run-ovn-kubernetes\") pod \"ovnkube-node-942xh\" (UID: \"31ff16f0-6054-4eba-942a-b07a5a7afac7\") " pod="openshift-ovn-kubernetes/ovnkube-node-942xh" Jul 07 15:41:02.636970 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.625005 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/57e72845-ffcd-4aa2-91c8-05d2f00b6b49-system-cni-dir\") pod \"multus-jzprp\" (UID: \"57e72845-ffcd-4aa2-91c8-05d2f00b6b49\") " pod="openshift-multus/multus-jzprp" Jul 07 15:41:02.636970 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.625026 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/57e72845-ffcd-4aa2-91c8-05d2f00b6b49-host-var-lib-cni-bin\") pod \"multus-jzprp\" (UID: \"57e72845-ffcd-4aa2-91c8-05d2f00b6b49\") " pod="openshift-multus/multus-jzprp" Jul 07 15:41:02.636970 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.625077 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/76d2d5c2-57b5-4c2d-81f9-73f49cde5fed-etc-selinux\") pod \"aws-ebs-csi-driver-node-s8x25\" (UID: \"76d2d5c2-57b5-4c2d-81f9-73f49cde5fed\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8x25" Jul 07 15:41:02.636970 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.625108 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdz5m\" (UniqueName: \"kubernetes.io/projected/76d2d5c2-57b5-4c2d-81f9-73f49cde5fed-kube-api-access-tdz5m\") pod \"aws-ebs-csi-driver-node-s8x25\" (UID: \"76d2d5c2-57b5-4c2d-81f9-73f49cde5fed\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8x25" Jul 07 15:41:02.637817 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.625133 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9l86g\" (UniqueName: \"kubernetes.io/projected/d6c50a3b-0739-431c-80b0-8f42c4b00405-kube-api-access-9l86g\") pod \"node-resolver-qggz6\" (UID: \"d6c50a3b-0739-431c-80b0-8f42c4b00405\") " pod="openshift-dns/node-resolver-qggz6" Jul 07 15:41:02.637817 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.625167 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d3ca04a5-3622-461f-9b02-0220e1c17814-tuning-conf-dir\") pod \"multus-additional-cni-plugins-f7x65\" (UID: \"d3ca04a5-3622-461f-9b02-0220e1c17814\") " pod="openshift-multus/multus-additional-cni-plugins-f7x65" Jul 07 15:41:02.637817 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.625189 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4e14c00d-b605-4eee-aaa1-8b07158de8f0-etc-sysconfig\") pod \"tuned-4glxt\" (UID: \"4e14c00d-b605-4eee-aaa1-8b07158de8f0\") " pod="openshift-cluster-node-tuning-operator/tuned-4glxt" Jul 07 15:41:02.637817 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.625208 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4e14c00d-b605-4eee-aaa1-8b07158de8f0-host\") pod \"tuned-4glxt\" (UID: \"4e14c00d-b605-4eee-aaa1-8b07158de8f0\") " pod="openshift-cluster-node-tuning-operator/tuned-4glxt" Jul 07 15:41:02.637817 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.625228 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/57e72845-ffcd-4aa2-91c8-05d2f00b6b49-host-run-k8s-cni-cncf-io\") pod \"multus-jzprp\" (UID: \"57e72845-ffcd-4aa2-91c8-05d2f00b6b49\") " pod="openshift-multus/multus-jzprp" Jul 07 15:41:02.637817 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.625248 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/57e72845-ffcd-4aa2-91c8-05d2f00b6b49-host-run-netns\") pod \"multus-jzprp\" (UID: \"57e72845-ffcd-4aa2-91c8-05d2f00b6b49\") " pod="openshift-multus/multus-jzprp" Jul 07 15:41:02.637817 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.625266 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/57e72845-ffcd-4aa2-91c8-05d2f00b6b49-host-run-multus-certs\") pod \"multus-jzprp\" (UID: \"57e72845-ffcd-4aa2-91c8-05d2f00b6b49\") " pod="openshift-multus/multus-jzprp" Jul 07 15:41:02.637817 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.625286 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6brw\" (UniqueName: \"kubernetes.io/projected/e1f2bb91-99ac-4353-b790-622cb6d40663-kube-api-access-n6brw\") pod \"network-check-target-94vlx\" (UID: \"e1f2bb91-99ac-4353-b790-622cb6d40663\") " pod="openshift-network-diagnostics/network-check-target-94vlx" Jul 07 15:41:02.637817 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.625307 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ec51f9ec-f9f7-4390-9954-eb4ab30b8c4d-host-slash\") pod \"iptables-alerter-lrpv9\" (UID: \"ec51f9ec-f9f7-4390-9954-eb4ab30b8c4d\") " pod="openshift-network-operator/iptables-alerter-lrpv9" Jul 07 15:41:02.637817 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.625327 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4e14c00d-b605-4eee-aaa1-8b07158de8f0-etc-modprobe-d\") pod \"tuned-4glxt\" (UID: \"4e14c00d-b605-4eee-aaa1-8b07158de8f0\") " pod="openshift-cluster-node-tuning-operator/tuned-4glxt" Jul 07 15:41:02.637817 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.625347 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6c50a3b-0739-431c-80b0-8f42c4b00405-tmp-dir\") pod \"node-resolver-qggz6\" (UID: \"d6c50a3b-0739-431c-80b0-8f42c4b00405\") " pod="openshift-dns/node-resolver-qggz6" Jul 07 15:41:02.637817 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.625366 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d3ca04a5-3622-461f-9b02-0220e1c17814-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-f7x65\" (UID: \"d3ca04a5-3622-461f-9b02-0220e1c17814\") " pod="openshift-multus/multus-additional-cni-plugins-f7x65" Jul 07 15:41:02.637817 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.625387 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d3ca04a5-3622-461f-9b02-0220e1c17814-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-f7x65\" (UID: \"d3ca04a5-3622-461f-9b02-0220e1c17814\") " pod="openshift-multus/multus-additional-cni-plugins-f7x65" Jul 07 15:41:02.638260 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.625409 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4e14c00d-b605-4eee-aaa1-8b07158de8f0-etc-kubernetes\") pod \"tuned-4glxt\" (UID: \"4e14c00d-b605-4eee-aaa1-8b07158de8f0\") " pod="openshift-cluster-node-tuning-operator/tuned-4glxt" Jul 07 15:41:02.638260 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.625431 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxxw4\" (UniqueName: \"kubernetes.io/projected/e5c3e573-02fc-4e63-b475-65ee8b00f34b-kube-api-access-nxxw4\") pod \"node-ca-9fmf8\" (UID: \"e5c3e573-02fc-4e63-b475-65ee8b00f34b\") " pod="openshift-image-registry/node-ca-9fmf8" Jul 07 15:41:02.638260 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.625459 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/31ff16f0-6054-4eba-942a-b07a5a7afac7-var-lib-openvswitch\") pod \"ovnkube-node-942xh\" (UID: \"31ff16f0-6054-4eba-942a-b07a5a7afac7\") " pod="openshift-ovn-kubernetes/ovnkube-node-942xh" Jul 07 15:41:02.638260 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.625481 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/31ff16f0-6054-4eba-942a-b07a5a7afac7-log-socket\") pod \"ovnkube-node-942xh\" (UID: \"31ff16f0-6054-4eba-942a-b07a5a7afac7\") " pod="openshift-ovn-kubernetes/ovnkube-node-942xh" Jul 07 15:41:02.638260 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.625515 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/31ff16f0-6054-4eba-942a-b07a5a7afac7-host-cni-netd\") pod \"ovnkube-node-942xh\" (UID: \"31ff16f0-6054-4eba-942a-b07a5a7afac7\") " pod="openshift-ovn-kubernetes/ovnkube-node-942xh" Jul 07 15:41:02.638260 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.625533 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/57e72845-ffcd-4aa2-91c8-05d2f00b6b49-hostroot\") pod \"multus-jzprp\" (UID: \"57e72845-ffcd-4aa2-91c8-05d2f00b6b49\") " pod="openshift-multus/multus-jzprp" Jul 07 15:41:02.638260 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.625551 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d3ca04a5-3622-461f-9b02-0220e1c17814-system-cni-dir\") pod \"multus-additional-cni-plugins-f7x65\" (UID: \"d3ca04a5-3622-461f-9b02-0220e1c17814\") " pod="openshift-multus/multus-additional-cni-plugins-f7x65" Jul 07 15:41:02.638260 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.625570 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4e14c00d-b605-4eee-aaa1-8b07158de8f0-etc-systemd\") pod \"tuned-4glxt\" (UID: \"4e14c00d-b605-4eee-aaa1-8b07158de8f0\") " pod="openshift-cluster-node-tuning-operator/tuned-4glxt" Jul 07 15:41:02.638260 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.625587 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4e14c00d-b605-4eee-aaa1-8b07158de8f0-run\") pod \"tuned-4glxt\" (UID: \"4e14c00d-b605-4eee-aaa1-8b07158de8f0\") " pod="openshift-cluster-node-tuning-operator/tuned-4glxt" Jul 07 15:41:02.638260 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.625604 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4e14c00d-b605-4eee-aaa1-8b07158de8f0-sys\") pod \"tuned-4glxt\" (UID: \"4e14c00d-b605-4eee-aaa1-8b07158de8f0\") " pod="openshift-cluster-node-tuning-operator/tuned-4glxt" Jul 07 15:41:02.638260 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.625632 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/31ff16f0-6054-4eba-942a-b07a5a7afac7-etc-openvswitch\") pod \"ovnkube-node-942xh\" (UID: \"31ff16f0-6054-4eba-942a-b07a5a7afac7\") " pod="openshift-ovn-kubernetes/ovnkube-node-942xh" Jul 07 15:41:02.638260 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.625655 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/31ff16f0-6054-4eba-942a-b07a5a7afac7-run-ovn\") pod \"ovnkube-node-942xh\" (UID: \"31ff16f0-6054-4eba-942a-b07a5a7afac7\") " pod="openshift-ovn-kubernetes/ovnkube-node-942xh" Jul 07 15:41:02.638260 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.625677 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rk5r\" (UniqueName: \"kubernetes.io/projected/4e14c00d-b605-4eee-aaa1-8b07158de8f0-kube-api-access-7rk5r\") pod \"tuned-4glxt\" (UID: \"4e14c00d-b605-4eee-aaa1-8b07158de8f0\") " pod="openshift-cluster-node-tuning-operator/tuned-4glxt" Jul 07 15:41:02.638260 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.625706 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/31ff16f0-6054-4eba-942a-b07a5a7afac7-run-openvswitch\") pod \"ovnkube-node-942xh\" (UID: \"31ff16f0-6054-4eba-942a-b07a5a7afac7\") " pod="openshift-ovn-kubernetes/ovnkube-node-942xh" Jul 07 15:41:02.638741 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.625726 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/31ff16f0-6054-4eba-942a-b07a5a7afac7-host-cni-bin\") pod \"ovnkube-node-942xh\" (UID: \"31ff16f0-6054-4eba-942a-b07a5a7afac7\") " pod="openshift-ovn-kubernetes/ovnkube-node-942xh" Jul 07 15:41:02.638741 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.625758 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5dsj\" (UniqueName: \"kubernetes.io/projected/d3ca04a5-3622-461f-9b02-0220e1c17814-kube-api-access-x5dsj\") pod \"multus-additional-cni-plugins-f7x65\" (UID: \"d3ca04a5-3622-461f-9b02-0220e1c17814\") " pod="openshift-multus/multus-additional-cni-plugins-f7x65" Jul 07 15:41:02.638741 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.625780 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4e14c00d-b605-4eee-aaa1-8b07158de8f0-etc-sysctl-conf\") pod \"tuned-4glxt\" (UID: \"4e14c00d-b605-4eee-aaa1-8b07158de8f0\") " pod="openshift-cluster-node-tuning-operator/tuned-4glxt" Jul 07 15:41:02.638741 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.625803 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/31ff16f0-6054-4eba-942a-b07a5a7afac7-ovn-node-metrics-cert\") pod \"ovnkube-node-942xh\" (UID: \"31ff16f0-6054-4eba-942a-b07a5a7afac7\") " pod="openshift-ovn-kubernetes/ovnkube-node-942xh" Jul 07 15:41:02.638741 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.625822 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/57e72845-ffcd-4aa2-91c8-05d2f00b6b49-multus-cni-dir\") pod \"multus-jzprp\" (UID: \"57e72845-ffcd-4aa2-91c8-05d2f00b6b49\") " pod="openshift-multus/multus-jzprp" Jul 07 15:41:02.638741 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.625841 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/57e72845-ffcd-4aa2-91c8-05d2f00b6b49-cnibin\") pod \"multus-jzprp\" (UID: \"57e72845-ffcd-4aa2-91c8-05d2f00b6b49\") " pod="openshift-multus/multus-jzprp" Jul 07 15:41:02.638741 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.625861 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/57e72845-ffcd-4aa2-91c8-05d2f00b6b49-multus-conf-dir\") pod \"multus-jzprp\" (UID: \"57e72845-ffcd-4aa2-91c8-05d2f00b6b49\") " pod="openshift-multus/multus-jzprp" Jul 07 15:41:02.638741 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.625880 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/31ff16f0-6054-4eba-942a-b07a5a7afac7-systemd-units\") pod \"ovnkube-node-942xh\" (UID: \"31ff16f0-6054-4eba-942a-b07a5a7afac7\") " pod="openshift-ovn-kubernetes/ovnkube-node-942xh" Jul 07 15:41:02.638741 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.625902 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0c5071d6-cead-4992-a7e9-9a752b955cae-kubelet-config\") pod \"global-pull-secret-syncer-6qzkn\" (UID: \"0c5071d6-cead-4992-a7e9-9a752b955cae\") " pod="kube-system/global-pull-secret-syncer-6qzkn" Jul 07 15:41:02.638741 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.625923 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/57e72845-ffcd-4aa2-91c8-05d2f00b6b49-multus-socket-dir-parent\") pod \"multus-jzprp\" (UID: \"57e72845-ffcd-4aa2-91c8-05d2f00b6b49\") " pod="openshift-multus/multus-jzprp" Jul 07 15:41:02.638741 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.625955 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/76d2d5c2-57b5-4c2d-81f9-73f49cde5fed-device-dir\") pod \"aws-ebs-csi-driver-node-s8x25\" (UID: \"76d2d5c2-57b5-4c2d-81f9-73f49cde5fed\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8x25" Jul 07 15:41:02.638741 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.625986 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d6c50a3b-0739-431c-80b0-8f42c4b00405-hosts-file\") pod \"node-resolver-qggz6\" (UID: \"d6c50a3b-0739-431c-80b0-8f42c4b00405\") " pod="openshift-dns/node-resolver-qggz6" Jul 07 15:41:02.638741 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.626018 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/57e72845-ffcd-4aa2-91c8-05d2f00b6b49-os-release\") pod \"multus-jzprp\" (UID: \"57e72845-ffcd-4aa2-91c8-05d2f00b6b49\") " pod="openshift-multus/multus-jzprp" Jul 07 15:41:02.638741 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.626040 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d3ca04a5-3622-461f-9b02-0220e1c17814-cnibin\") pod \"multus-additional-cni-plugins-f7x65\" (UID: \"d3ca04a5-3622-461f-9b02-0220e1c17814\") " pod="openshift-multus/multus-additional-cni-plugins-f7x65" Jul 07 15:41:02.639250 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.626055 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/31ff16f0-6054-4eba-942a-b07a5a7afac7-host-run-netns\") pod \"ovnkube-node-942xh\" (UID: \"31ff16f0-6054-4eba-942a-b07a5a7afac7\") " pod="openshift-ovn-kubernetes/ovnkube-node-942xh" Jul 07 15:41:02.639250 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.626071 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/57e72845-ffcd-4aa2-91c8-05d2f00b6b49-etc-kubernetes\") pod \"multus-jzprp\" (UID: \"57e72845-ffcd-4aa2-91c8-05d2f00b6b49\") " pod="openshift-multus/multus-jzprp" Jul 07 15:41:02.639250 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.626094 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/76d2d5c2-57b5-4c2d-81f9-73f49cde5fed-socket-dir\") pod \"aws-ebs-csi-driver-node-s8x25\" (UID: \"76d2d5c2-57b5-4c2d-81f9-73f49cde5fed\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8x25" Jul 07 15:41:02.639250 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.626112 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d3ca04a5-3622-461f-9b02-0220e1c17814-cni-binary-copy\") pod \"multus-additional-cni-plugins-f7x65\" (UID: \"d3ca04a5-3622-461f-9b02-0220e1c17814\") " pod="openshift-multus/multus-additional-cni-plugins-f7x65" Jul 07 15:41:02.639250 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.626127 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kmppq\" (UniqueName: \"kubernetes.io/projected/301c1027-169c-47f5-a3ea-89cde78790d2-kube-api-access-kmppq\") pod \"network-metrics-daemon-sdzxq\" (UID: \"301c1027-169c-47f5-a3ea-89cde78790d2\") " pod="openshift-multus/network-metrics-daemon-sdzxq" Jul 07 15:41:02.639250 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.626140 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4e14c00d-b605-4eee-aaa1-8b07158de8f0-etc-sysctl-d\") pod \"tuned-4glxt\" (UID: \"4e14c00d-b605-4eee-aaa1-8b07158de8f0\") " pod="openshift-cluster-node-tuning-operator/tuned-4glxt" Jul 07 15:41:02.639250 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.626159 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0c5071d6-cead-4992-a7e9-9a752b955cae-original-pull-secret\") pod \"global-pull-secret-syncer-6qzkn\" (UID: \"0c5071d6-cead-4992-a7e9-9a752b955cae\") " pod="kube-system/global-pull-secret-syncer-6qzkn" Jul 07 15:41:02.639250 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.626182 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/57e72845-ffcd-4aa2-91c8-05d2f00b6b49-host-var-lib-cni-multus\") pod \"multus-jzprp\" (UID: \"57e72845-ffcd-4aa2-91c8-05d2f00b6b49\") " pod="openshift-multus/multus-jzprp" Jul 07 15:41:02.639250 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.626203 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/57e72845-ffcd-4aa2-91c8-05d2f00b6b49-multus-daemon-config\") pod \"multus-jzprp\" (UID: \"57e72845-ffcd-4aa2-91c8-05d2f00b6b49\") " pod="openshift-multus/multus-jzprp" Jul 07 15:41:02.639250 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.626222 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/76d2d5c2-57b5-4c2d-81f9-73f49cde5fed-sys-fs\") pod \"aws-ebs-csi-driver-node-s8x25\" (UID: \"76d2d5c2-57b5-4c2d-81f9-73f49cde5fed\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8x25" Jul 07 15:41:02.639250 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.626246 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/76d2d5c2-57b5-4c2d-81f9-73f49cde5fed-kubelet-dir\") pod \"aws-ebs-csi-driver-node-s8x25\" (UID: \"76d2d5c2-57b5-4c2d-81f9-73f49cde5fed\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8x25" Jul 07 15:41:02.639250 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.626268 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/d3ca04a5-3622-461f-9b02-0220e1c17814-os-release\") pod \"multus-additional-cni-plugins-f7x65\" (UID: \"d3ca04a5-3622-461f-9b02-0220e1c17814\") " pod="openshift-multus/multus-additional-cni-plugins-f7x65" Jul 07 15:41:02.639250 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.626295 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/31ff16f0-6054-4eba-942a-b07a5a7afac7-host-slash\") pod \"ovnkube-node-942xh\" (UID: \"31ff16f0-6054-4eba-942a-b07a5a7afac7\") " pod="openshift-ovn-kubernetes/ovnkube-node-942xh" Jul 07 15:41:02.639738 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.626316 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/31ff16f0-6054-4eba-942a-b07a5a7afac7-node-log\") pod \"ovnkube-node-942xh\" (UID: \"31ff16f0-6054-4eba-942a-b07a5a7afac7\") " pod="openshift-ovn-kubernetes/ovnkube-node-942xh" Jul 07 15:41:02.639738 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.626336 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/31ff16f0-6054-4eba-942a-b07a5a7afac7-env-overrides\") pod \"ovnkube-node-942xh\" (UID: \"31ff16f0-6054-4eba-942a-b07a5a7afac7\") " pod="openshift-ovn-kubernetes/ovnkube-node-942xh" Jul 07 15:41:02.639738 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.626359 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/57e72845-ffcd-4aa2-91c8-05d2f00b6b49-cni-binary-copy\") pod \"multus-jzprp\" (UID: \"57e72845-ffcd-4aa2-91c8-05d2f00b6b49\") " pod="openshift-multus/multus-jzprp" Jul 07 15:41:02.639738 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.626384 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4e14c00d-b605-4eee-aaa1-8b07158de8f0-var-lib-kubelet\") pod \"tuned-4glxt\" (UID: \"4e14c00d-b605-4eee-aaa1-8b07158de8f0\") " pod="openshift-cluster-node-tuning-operator/tuned-4glxt" Jul 07 15:41:02.745345 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.744941 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/301c1027-169c-47f5-a3ea-89cde78790d2-metrics-certs\") pod \"network-metrics-daemon-sdzxq\" (UID: \"301c1027-169c-47f5-a3ea-89cde78790d2\") " pod="openshift-multus/network-metrics-daemon-sdzxq" Jul 07 15:41:02.745345 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.744985 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4e14c00d-b605-4eee-aaa1-8b07158de8f0-lib-modules\") pod \"tuned-4glxt\" (UID: \"4e14c00d-b605-4eee-aaa1-8b07158de8f0\") " pod="openshift-cluster-node-tuning-operator/tuned-4glxt" Jul 07 15:41:02.745345 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.745019 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/31ff16f0-6054-4eba-942a-b07a5a7afac7-ovnkube-script-lib\") pod \"ovnkube-node-942xh\" (UID: \"31ff16f0-6054-4eba-942a-b07a5a7afac7\") " pod="openshift-ovn-kubernetes/ovnkube-node-942xh" Jul 07 15:41:02.745345 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.745070 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzqjv\" (UniqueName: \"kubernetes.io/projected/31ff16f0-6054-4eba-942a-b07a5a7afac7-kube-api-access-lzqjv\") pod \"ovnkube-node-942xh\" (UID: \"31ff16f0-6054-4eba-942a-b07a5a7afac7\") " pod="openshift-ovn-kubernetes/ovnkube-node-942xh" Jul 07 15:41:02.745345 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.745098 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0c5071d6-cead-4992-a7e9-9a752b955cae-dbus\") pod \"global-pull-secret-syncer-6qzkn\" (UID: \"0c5071d6-cead-4992-a7e9-9a752b955cae\") " pod="kube-system/global-pull-secret-syncer-6qzkn" Jul 07 15:41:02.745345 ip-10-0-132-65 kubenswrapper[2565]: E0707 15:41:02.745107 2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 15:41:02.745345 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.745123 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e6634bed-1f16-4667-ae9e-c08984716ce5-konnectivity-ca\") pod \"konnectivity-agent-m4wtp\" (UID: \"e6634bed-1f16-4667-ae9e-c08984716ce5\") " pod="kube-system/konnectivity-agent-m4wtp" Jul 07 15:41:02.745345 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.745149 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/76d2d5c2-57b5-4c2d-81f9-73f49cde5fed-registration-dir\") pod \"aws-ebs-csi-driver-node-s8x25\" (UID: \"76d2d5c2-57b5-4c2d-81f9-73f49cde5fed\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8x25" Jul 07 15:41:02.745345 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.745172 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4e14c00d-b605-4eee-aaa1-8b07158de8f0-lib-modules\") pod \"tuned-4glxt\" (UID: \"4e14c00d-b605-4eee-aaa1-8b07158de8f0\") " pod="openshift-cluster-node-tuning-operator/tuned-4glxt" Jul 07 15:41:02.745345 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.745286 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e5c3e573-02fc-4e63-b475-65ee8b00f34b-host\") pod \"node-ca-9fmf8\" (UID: \"e5c3e573-02fc-4e63-b475-65ee8b00f34b\") " pod="openshift-image-registry/node-ca-9fmf8" Jul 07 15:41:02.745345 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.745317 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/31ff16f0-6054-4eba-942a-b07a5a7afac7-host-run-ovn-kubernetes\") pod \"ovnkube-node-942xh\" (UID: \"31ff16f0-6054-4eba-942a-b07a5a7afac7\") " pod="openshift-ovn-kubernetes/ovnkube-node-942xh" Jul 07 15:41:02.745345 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.745343 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/57e72845-ffcd-4aa2-91c8-05d2f00b6b49-system-cni-dir\") pod \"multus-jzprp\" (UID: \"57e72845-ffcd-4aa2-91c8-05d2f00b6b49\") " pod="openshift-multus/multus-jzprp" Jul 07 15:41:02.746071 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.745370 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/57e72845-ffcd-4aa2-91c8-05d2f00b6b49-host-var-lib-cni-bin\") pod \"multus-jzprp\" (UID: \"57e72845-ffcd-4aa2-91c8-05d2f00b6b49\") " pod="openshift-multus/multus-jzprp" Jul 07 15:41:02.746071 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.745398 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/76d2d5c2-57b5-4c2d-81f9-73f49cde5fed-etc-selinux\") pod \"aws-ebs-csi-driver-node-s8x25\" (UID: \"76d2d5c2-57b5-4c2d-81f9-73f49cde5fed\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8x25" Jul 07 15:41:02.746071 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.745425 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdz5m\" (UniqueName: \"kubernetes.io/projected/76d2d5c2-57b5-4c2d-81f9-73f49cde5fed-kube-api-access-tdz5m\") pod \"aws-ebs-csi-driver-node-s8x25\" (UID: \"76d2d5c2-57b5-4c2d-81f9-73f49cde5fed\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8x25" Jul 07 15:41:02.746071 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.745455 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9l86g\" (UniqueName: \"kubernetes.io/projected/d6c50a3b-0739-431c-80b0-8f42c4b00405-kube-api-access-9l86g\") pod \"node-resolver-qggz6\" (UID: \"d6c50a3b-0739-431c-80b0-8f42c4b00405\") " pod="openshift-dns/node-resolver-qggz6" Jul 07 15:41:02.746071 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.745582 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/57e72845-ffcd-4aa2-91c8-05d2f00b6b49-host-var-lib-cni-bin\") pod \"multus-jzprp\" (UID: \"57e72845-ffcd-4aa2-91c8-05d2f00b6b49\") " pod="openshift-multus/multus-jzprp" Jul 07 15:41:02.746071 ip-10-0-132-65 kubenswrapper[2565]: E0707 15:41:02.745797 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/301c1027-169c-47f5-a3ea-89cde78790d2-metrics-certs podName:301c1027-169c-47f5-a3ea-89cde78790d2 nodeName:}" failed. No retries permitted until 2026-07-07 15:41:03.245768298 +0000 UTC m=+2.202345749 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/301c1027-169c-47f5-a3ea-89cde78790d2-metrics-certs") pod "network-metrics-daemon-sdzxq" (UID: "301c1027-169c-47f5-a3ea-89cde78790d2") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 15:41:02.746071 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.745870 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/76d2d5c2-57b5-4c2d-81f9-73f49cde5fed-etc-selinux\") pod \"aws-ebs-csi-driver-node-s8x25\" (UID: \"76d2d5c2-57b5-4c2d-81f9-73f49cde5fed\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8x25" Jul 07 15:41:02.746071 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.745882 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/57e72845-ffcd-4aa2-91c8-05d2f00b6b49-system-cni-dir\") pod \"multus-jzprp\" (UID: \"57e72845-ffcd-4aa2-91c8-05d2f00b6b49\") " pod="openshift-multus/multus-jzprp" Jul 07 15:41:02.746071 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.745853 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/31ff16f0-6054-4eba-942a-b07a5a7afac7-host-run-ovn-kubernetes\") pod \"ovnkube-node-942xh\" (UID: \"31ff16f0-6054-4eba-942a-b07a5a7afac7\") " pod="openshift-ovn-kubernetes/ovnkube-node-942xh" Jul 07 15:41:02.832021 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.746076 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e6634bed-1f16-4667-ae9e-c08984716ce5-konnectivity-ca\") pod \"konnectivity-agent-m4wtp\" (UID: \"e6634bed-1f16-4667-ae9e-c08984716ce5\") " pod="kube-system/konnectivity-agent-m4wtp" Jul 07 15:41:02.832021 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.746081 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e5c3e573-02fc-4e63-b475-65ee8b00f34b-host\") pod \"node-ca-9fmf8\" (UID: \"e5c3e573-02fc-4e63-b475-65ee8b00f34b\") " pod="openshift-image-registry/node-ca-9fmf8" Jul 07 15:41:02.832021 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.746130 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/76d2d5c2-57b5-4c2d-81f9-73f49cde5fed-registration-dir\") pod \"aws-ebs-csi-driver-node-s8x25\" (UID: \"76d2d5c2-57b5-4c2d-81f9-73f49cde5fed\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8x25" Jul 07 15:41:02.832021 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.746172 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/31ff16f0-6054-4eba-942a-b07a5a7afac7-ovnkube-script-lib\") pod \"ovnkube-node-942xh\" (UID: \"31ff16f0-6054-4eba-942a-b07a5a7afac7\") " pod="openshift-ovn-kubernetes/ovnkube-node-942xh" Jul 07 15:41:02.832021 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.746250 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d3ca04a5-3622-461f-9b02-0220e1c17814-tuning-conf-dir\") pod \"multus-additional-cni-plugins-f7x65\" (UID: \"d3ca04a5-3622-461f-9b02-0220e1c17814\") " pod="openshift-multus/multus-additional-cni-plugins-f7x65" Jul 07 15:41:02.832021 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.746316 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0c5071d6-cead-4992-a7e9-9a752b955cae-dbus\") pod \"global-pull-secret-syncer-6qzkn\" (UID: \"0c5071d6-cead-4992-a7e9-9a752b955cae\") " pod="kube-system/global-pull-secret-syncer-6qzkn" Jul 07 15:41:02.832021 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.746320 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4e14c00d-b605-4eee-aaa1-8b07158de8f0-etc-sysconfig\") pod \"tuned-4glxt\" (UID: \"4e14c00d-b605-4eee-aaa1-8b07158de8f0\") " pod="openshift-cluster-node-tuning-operator/tuned-4glxt" Jul 07 15:41:02.832021 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.746399 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4e14c00d-b605-4eee-aaa1-8b07158de8f0-host\") pod \"tuned-4glxt\" (UID: \"4e14c00d-b605-4eee-aaa1-8b07158de8f0\") " pod="openshift-cluster-node-tuning-operator/tuned-4glxt" Jul 07 15:41:02.832021 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.746409 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d3ca04a5-3622-461f-9b02-0220e1c17814-tuning-conf-dir\") pod \"multus-additional-cni-plugins-f7x65\" (UID: \"d3ca04a5-3622-461f-9b02-0220e1c17814\") " pod="openshift-multus/multus-additional-cni-plugins-f7x65" Jul 07 15:41:02.832021 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.746423 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/57e72845-ffcd-4aa2-91c8-05d2f00b6b49-host-run-k8s-cni-cncf-io\") pod \"multus-jzprp\" (UID: \"57e72845-ffcd-4aa2-91c8-05d2f00b6b49\") " pod="openshift-multus/multus-jzprp" Jul 07 15:41:02.832021 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.746447 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/57e72845-ffcd-4aa2-91c8-05d2f00b6b49-host-run-netns\") pod \"multus-jzprp\" (UID: \"57e72845-ffcd-4aa2-91c8-05d2f00b6b49\") " pod="openshift-multus/multus-jzprp" Jul 07 15:41:02.832021 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.746470 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/57e72845-ffcd-4aa2-91c8-05d2f00b6b49-host-run-multus-certs\") pod \"multus-jzprp\" (UID: \"57e72845-ffcd-4aa2-91c8-05d2f00b6b49\") " pod="openshift-multus/multus-jzprp" Jul 07 15:41:02.832021 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.746475 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4e14c00d-b605-4eee-aaa1-8b07158de8f0-etc-sysconfig\") pod \"tuned-4glxt\" (UID: \"4e14c00d-b605-4eee-aaa1-8b07158de8f0\") " pod="openshift-cluster-node-tuning-operator/tuned-4glxt" Jul 07 15:41:02.832021 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.746541 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6brw\" (UniqueName: \"kubernetes.io/projected/e1f2bb91-99ac-4353-b790-622cb6d40663-kube-api-access-n6brw\") pod \"network-check-target-94vlx\" (UID: \"e1f2bb91-99ac-4353-b790-622cb6d40663\") " pod="openshift-network-diagnostics/network-check-target-94vlx" Jul 07 15:41:02.832021 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.746562 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/57e72845-ffcd-4aa2-91c8-05d2f00b6b49-host-run-netns\") pod \"multus-jzprp\" (UID: \"57e72845-ffcd-4aa2-91c8-05d2f00b6b49\") " pod="openshift-multus/multus-jzprp" Jul 07 15:41:02.833057 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.746566 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/57e72845-ffcd-4aa2-91c8-05d2f00b6b49-host-run-multus-certs\") pod \"multus-jzprp\" (UID: \"57e72845-ffcd-4aa2-91c8-05d2f00b6b49\") " pod="openshift-multus/multus-jzprp" Jul 07 15:41:02.833057 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.746575 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ec51f9ec-f9f7-4390-9954-eb4ab30b8c4d-host-slash\") pod \"iptables-alerter-lrpv9\" (UID: \"ec51f9ec-f9f7-4390-9954-eb4ab30b8c4d\") " pod="openshift-network-operator/iptables-alerter-lrpv9" Jul 07 15:41:02.833057 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.746582 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/57e72845-ffcd-4aa2-91c8-05d2f00b6b49-host-run-k8s-cni-cncf-io\") pod \"multus-jzprp\" (UID: \"57e72845-ffcd-4aa2-91c8-05d2f00b6b49\") " pod="openshift-multus/multus-jzprp" Jul 07 15:41:02.833057 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.746615 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4e14c00d-b605-4eee-aaa1-8b07158de8f0-etc-modprobe-d\") pod \"tuned-4glxt\" (UID: \"4e14c00d-b605-4eee-aaa1-8b07158de8f0\") " pod="openshift-cluster-node-tuning-operator/tuned-4glxt" Jul 07 15:41:02.833057 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.746626 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4e14c00d-b605-4eee-aaa1-8b07158de8f0-host\") pod \"tuned-4glxt\" (UID: \"4e14c00d-b605-4eee-aaa1-8b07158de8f0\") " pod="openshift-cluster-node-tuning-operator/tuned-4glxt" Jul 07 15:41:02.833057 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.746792 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6c50a3b-0739-431c-80b0-8f42c4b00405-tmp-dir\") pod \"node-resolver-qggz6\" (UID: \"d6c50a3b-0739-431c-80b0-8f42c4b00405\") " pod="openshift-dns/node-resolver-qggz6" Jul 07 15:41:02.833057 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.746793 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ec51f9ec-f9f7-4390-9954-eb4ab30b8c4d-host-slash\") pod \"iptables-alerter-lrpv9\" (UID: \"ec51f9ec-f9f7-4390-9954-eb4ab30b8c4d\") " pod="openshift-network-operator/iptables-alerter-lrpv9" Jul 07 15:41:02.833057 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.746821 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4e14c00d-b605-4eee-aaa1-8b07158de8f0-etc-modprobe-d\") pod \"tuned-4glxt\" (UID: \"4e14c00d-b605-4eee-aaa1-8b07158de8f0\") " pod="openshift-cluster-node-tuning-operator/tuned-4glxt" Jul 07 15:41:02.833057 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.746848 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d3ca04a5-3622-461f-9b02-0220e1c17814-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-f7x65\" (UID: \"d3ca04a5-3622-461f-9b02-0220e1c17814\") " pod="openshift-multus/multus-additional-cni-plugins-f7x65" Jul 07 15:41:02.833057 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.746879 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d3ca04a5-3622-461f-9b02-0220e1c17814-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-f7x65\" (UID: \"d3ca04a5-3622-461f-9b02-0220e1c17814\") " pod="openshift-multus/multus-additional-cni-plugins-f7x65" Jul 07 15:41:02.833057 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.746906 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4e14c00d-b605-4eee-aaa1-8b07158de8f0-etc-kubernetes\") pod \"tuned-4glxt\" (UID: \"4e14c00d-b605-4eee-aaa1-8b07158de8f0\") " pod="openshift-cluster-node-tuning-operator/tuned-4glxt" Jul 07 15:41:02.833057 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.746948 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxxw4\" (UniqueName: \"kubernetes.io/projected/e5c3e573-02fc-4e63-b475-65ee8b00f34b-kube-api-access-nxxw4\") pod \"node-ca-9fmf8\" (UID: \"e5c3e573-02fc-4e63-b475-65ee8b00f34b\") " pod="openshift-image-registry/node-ca-9fmf8" Jul 07 15:41:02.833057 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.746964 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/31ff16f0-6054-4eba-942a-b07a5a7afac7-var-lib-openvswitch\") pod \"ovnkube-node-942xh\" (UID: \"31ff16f0-6054-4eba-942a-b07a5a7afac7\") " pod="openshift-ovn-kubernetes/ovnkube-node-942xh" Jul 07 15:41:02.833057 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.746978 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/31ff16f0-6054-4eba-942a-b07a5a7afac7-log-socket\") pod \"ovnkube-node-942xh\" (UID: \"31ff16f0-6054-4eba-942a-b07a5a7afac7\") " pod="openshift-ovn-kubernetes/ovnkube-node-942xh" Jul 07 15:41:02.835772 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.746997 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/31ff16f0-6054-4eba-942a-b07a5a7afac7-host-cni-netd\") pod \"ovnkube-node-942xh\" (UID: \"31ff16f0-6054-4eba-942a-b07a5a7afac7\") " pod="openshift-ovn-kubernetes/ovnkube-node-942xh" Jul 07 15:41:02.835772 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.747021 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6c50a3b-0739-431c-80b0-8f42c4b00405-tmp-dir\") pod \"node-resolver-qggz6\" (UID: \"d6c50a3b-0739-431c-80b0-8f42c4b00405\") " pod="openshift-dns/node-resolver-qggz6" Jul 07 15:41:02.835772 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.747020 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/57e72845-ffcd-4aa2-91c8-05d2f00b6b49-hostroot\") pod \"multus-jzprp\" (UID: \"57e72845-ffcd-4aa2-91c8-05d2f00b6b49\") " pod="openshift-multus/multus-jzprp" Jul 07 15:41:02.835772 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.747050 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4e14c00d-b605-4eee-aaa1-8b07158de8f0-etc-kubernetes\") pod \"tuned-4glxt\" (UID: \"4e14c00d-b605-4eee-aaa1-8b07158de8f0\") " pod="openshift-cluster-node-tuning-operator/tuned-4glxt" Jul 07 15:41:02.835772 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.747066 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d3ca04a5-3622-461f-9b02-0220e1c17814-system-cni-dir\") pod \"multus-additional-cni-plugins-f7x65\" (UID: \"d3ca04a5-3622-461f-9b02-0220e1c17814\") " pod="openshift-multus/multus-additional-cni-plugins-f7x65" Jul 07 15:41:02.835772 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.747102 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/31ff16f0-6054-4eba-942a-b07a5a7afac7-host-cni-netd\") pod \"ovnkube-node-942xh\" (UID: \"31ff16f0-6054-4eba-942a-b07a5a7afac7\") " pod="openshift-ovn-kubernetes/ovnkube-node-942xh" Jul 07 15:41:02.835772 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.747106 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/57e72845-ffcd-4aa2-91c8-05d2f00b6b49-hostroot\") pod \"multus-jzprp\" (UID: \"57e72845-ffcd-4aa2-91c8-05d2f00b6b49\") " pod="openshift-multus/multus-jzprp" Jul 07 15:41:02.835772 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.747092 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/31ff16f0-6054-4eba-942a-b07a5a7afac7-log-socket\") pod \"ovnkube-node-942xh\" (UID: \"31ff16f0-6054-4eba-942a-b07a5a7afac7\") " pod="openshift-ovn-kubernetes/ovnkube-node-942xh" Jul 07 15:41:02.835772 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.747134 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4e14c00d-b605-4eee-aaa1-8b07158de8f0-etc-systemd\") pod \"tuned-4glxt\" (UID: \"4e14c00d-b605-4eee-aaa1-8b07158de8f0\") " pod="openshift-cluster-node-tuning-operator/tuned-4glxt" Jul 07 15:41:02.835772 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.747178 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4e14c00d-b605-4eee-aaa1-8b07158de8f0-run\") pod \"tuned-4glxt\" (UID: \"4e14c00d-b605-4eee-aaa1-8b07158de8f0\") " pod="openshift-cluster-node-tuning-operator/tuned-4glxt" Jul 07 15:41:02.835772 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.747178 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4e14c00d-b605-4eee-aaa1-8b07158de8f0-etc-systemd\") pod \"tuned-4glxt\" (UID: \"4e14c00d-b605-4eee-aaa1-8b07158de8f0\") " pod="openshift-cluster-node-tuning-operator/tuned-4glxt" Jul 07 15:41:02.835772 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.747115 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d3ca04a5-3622-461f-9b02-0220e1c17814-system-cni-dir\") pod \"multus-additional-cni-plugins-f7x65\" (UID: \"d3ca04a5-3622-461f-9b02-0220e1c17814\") " pod="openshift-multus/multus-additional-cni-plugins-f7x65" Jul 07 15:41:02.835772 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.747206 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4e14c00d-b605-4eee-aaa1-8b07158de8f0-sys\") pod \"tuned-4glxt\" (UID: \"4e14c00d-b605-4eee-aaa1-8b07158de8f0\") " pod="openshift-cluster-node-tuning-operator/tuned-4glxt" Jul 07 15:41:02.835772 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.747230 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4e14c00d-b605-4eee-aaa1-8b07158de8f0-run\") pod \"tuned-4glxt\" (UID: \"4e14c00d-b605-4eee-aaa1-8b07158de8f0\") " pod="openshift-cluster-node-tuning-operator/tuned-4glxt" Jul 07 15:41:02.835772 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.747242 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/31ff16f0-6054-4eba-942a-b07a5a7afac7-etc-openvswitch\") pod \"ovnkube-node-942xh\" (UID: \"31ff16f0-6054-4eba-942a-b07a5a7afac7\") " pod="openshift-ovn-kubernetes/ovnkube-node-942xh" Jul 07 15:41:02.836413 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.747268 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/31ff16f0-6054-4eba-942a-b07a5a7afac7-run-ovn\") pod \"ovnkube-node-942xh\" (UID: \"31ff16f0-6054-4eba-942a-b07a5a7afac7\") " pod="openshift-ovn-kubernetes/ovnkube-node-942xh" Jul 07 15:41:02.836413 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.747285 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rk5r\" (UniqueName: \"kubernetes.io/projected/4e14c00d-b605-4eee-aaa1-8b07158de8f0-kube-api-access-7rk5r\") pod \"tuned-4glxt\" (UID: \"4e14c00d-b605-4eee-aaa1-8b07158de8f0\") " pod="openshift-cluster-node-tuning-operator/tuned-4glxt" Jul 07 15:41:02.836413 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.747286 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4e14c00d-b605-4eee-aaa1-8b07158de8f0-sys\") pod \"tuned-4glxt\" (UID: \"4e14c00d-b605-4eee-aaa1-8b07158de8f0\") " pod="openshift-cluster-node-tuning-operator/tuned-4glxt" Jul 07 15:41:02.836413 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.747320 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/31ff16f0-6054-4eba-942a-b07a5a7afac7-etc-openvswitch\") pod \"ovnkube-node-942xh\" (UID: \"31ff16f0-6054-4eba-942a-b07a5a7afac7\") " pod="openshift-ovn-kubernetes/ovnkube-node-942xh" Jul 07 15:41:02.836413 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.747329 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/31ff16f0-6054-4eba-942a-b07a5a7afac7-run-ovn\") pod \"ovnkube-node-942xh\" (UID: \"31ff16f0-6054-4eba-942a-b07a5a7afac7\") " pod="openshift-ovn-kubernetes/ovnkube-node-942xh" Jul 07 15:41:02.836413 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.747352 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/31ff16f0-6054-4eba-942a-b07a5a7afac7-run-openvswitch\") pod \"ovnkube-node-942xh\" (UID: \"31ff16f0-6054-4eba-942a-b07a5a7afac7\") " pod="openshift-ovn-kubernetes/ovnkube-node-942xh" Jul 07 15:41:02.836413 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.747379 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/31ff16f0-6054-4eba-942a-b07a5a7afac7-host-cni-bin\") pod \"ovnkube-node-942xh\" (UID: \"31ff16f0-6054-4eba-942a-b07a5a7afac7\") " pod="openshift-ovn-kubernetes/ovnkube-node-942xh" Jul 07 15:41:02.836413 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.747395 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/31ff16f0-6054-4eba-942a-b07a5a7afac7-run-openvswitch\") pod \"ovnkube-node-942xh\" (UID: \"31ff16f0-6054-4eba-942a-b07a5a7afac7\") " pod="openshift-ovn-kubernetes/ovnkube-node-942xh" Jul 07 15:41:02.836413 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.747404 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5dsj\" (UniqueName: \"kubernetes.io/projected/d3ca04a5-3622-461f-9b02-0220e1c17814-kube-api-access-x5dsj\") pod \"multus-additional-cni-plugins-f7x65\" (UID: \"d3ca04a5-3622-461f-9b02-0220e1c17814\") " pod="openshift-multus/multus-additional-cni-plugins-f7x65" Jul 07 15:41:02.836413 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.747447 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/31ff16f0-6054-4eba-942a-b07a5a7afac7-host-cni-bin\") pod \"ovnkube-node-942xh\" (UID: \"31ff16f0-6054-4eba-942a-b07a5a7afac7\") " pod="openshift-ovn-kubernetes/ovnkube-node-942xh" Jul 07 15:41:02.836413 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.747473 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d3ca04a5-3622-461f-9b02-0220e1c17814-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-f7x65\" (UID: \"d3ca04a5-3622-461f-9b02-0220e1c17814\") " pod="openshift-multus/multus-additional-cni-plugins-f7x65" Jul 07 15:41:02.836413 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.747471 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d3ca04a5-3622-461f-9b02-0220e1c17814-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-f7x65\" (UID: \"d3ca04a5-3622-461f-9b02-0220e1c17814\") " pod="openshift-multus/multus-additional-cni-plugins-f7x65" Jul 07 15:41:02.836413 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.747508 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4e14c00d-b605-4eee-aaa1-8b07158de8f0-etc-sysctl-conf\") pod \"tuned-4glxt\" (UID: \"4e14c00d-b605-4eee-aaa1-8b07158de8f0\") " pod="openshift-cluster-node-tuning-operator/tuned-4glxt" Jul 07 15:41:02.836413 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.747608 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/31ff16f0-6054-4eba-942a-b07a5a7afac7-ovn-node-metrics-cert\") pod \"ovnkube-node-942xh\" (UID: \"31ff16f0-6054-4eba-942a-b07a5a7afac7\") " pod="openshift-ovn-kubernetes/ovnkube-node-942xh" Jul 07 15:41:02.838478 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.747630 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/31ff16f0-6054-4eba-942a-b07a5a7afac7-var-lib-openvswitch\") pod \"ovnkube-node-942xh\" (UID: \"31ff16f0-6054-4eba-942a-b07a5a7afac7\") " pod="openshift-ovn-kubernetes/ovnkube-node-942xh" Jul 07 15:41:02.838478 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.747632 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4e14c00d-b605-4eee-aaa1-8b07158de8f0-etc-sysctl-conf\") pod \"tuned-4glxt\" (UID: \"4e14c00d-b605-4eee-aaa1-8b07158de8f0\") " pod="openshift-cluster-node-tuning-operator/tuned-4glxt" Jul 07 15:41:02.838478 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.747688 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/57e72845-ffcd-4aa2-91c8-05d2f00b6b49-multus-cni-dir\") pod \"multus-jzprp\" (UID: \"57e72845-ffcd-4aa2-91c8-05d2f00b6b49\") " pod="openshift-multus/multus-jzprp" Jul 07 15:41:02.838478 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.747722 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/57e72845-ffcd-4aa2-91c8-05d2f00b6b49-cnibin\") pod \"multus-jzprp\" (UID: \"57e72845-ffcd-4aa2-91c8-05d2f00b6b49\") " pod="openshift-multus/multus-jzprp" Jul 07 15:41:02.838478 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.747745 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/57e72845-ffcd-4aa2-91c8-05d2f00b6b49-multus-conf-dir\") pod \"multus-jzprp\" (UID: \"57e72845-ffcd-4aa2-91c8-05d2f00b6b49\") " pod="openshift-multus/multus-jzprp" Jul 07 15:41:02.838478 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.747770 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/31ff16f0-6054-4eba-942a-b07a5a7afac7-systemd-units\") pod \"ovnkube-node-942xh\" (UID: \"31ff16f0-6054-4eba-942a-b07a5a7afac7\") " pod="openshift-ovn-kubernetes/ovnkube-node-942xh" Jul 07 15:41:02.838478 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.747788 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0c5071d6-cead-4992-a7e9-9a752b955cae-kubelet-config\") pod \"global-pull-secret-syncer-6qzkn\" (UID: \"0c5071d6-cead-4992-a7e9-9a752b955cae\") " pod="kube-system/global-pull-secret-syncer-6qzkn" Jul 07 15:41:02.838478 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.747804 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/57e72845-ffcd-4aa2-91c8-05d2f00b6b49-multus-socket-dir-parent\") pod \"multus-jzprp\" (UID: \"57e72845-ffcd-4aa2-91c8-05d2f00b6b49\") " pod="openshift-multus/multus-jzprp" Jul 07 15:41:02.838478 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.747805 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/57e72845-ffcd-4aa2-91c8-05d2f00b6b49-cnibin\") pod \"multus-jzprp\" (UID: \"57e72845-ffcd-4aa2-91c8-05d2f00b6b49\") " pod="openshift-multus/multus-jzprp" Jul 07 15:41:02.838478 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.747772 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/57e72845-ffcd-4aa2-91c8-05d2f00b6b49-multus-conf-dir\") pod \"multus-jzprp\" (UID: \"57e72845-ffcd-4aa2-91c8-05d2f00b6b49\") " pod="openshift-multus/multus-jzprp" Jul 07 15:41:02.838478 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.747819 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/76d2d5c2-57b5-4c2d-81f9-73f49cde5fed-device-dir\") pod \"aws-ebs-csi-driver-node-s8x25\" (UID: \"76d2d5c2-57b5-4c2d-81f9-73f49cde5fed\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8x25" Jul 07 15:41:02.838478 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.747868 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/57e72845-ffcd-4aa2-91c8-05d2f00b6b49-multus-cni-dir\") pod \"multus-jzprp\" (UID: \"57e72845-ffcd-4aa2-91c8-05d2f00b6b49\") " pod="openshift-multus/multus-jzprp" Jul 07 15:41:02.838478 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.747867 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/31ff16f0-6054-4eba-942a-b07a5a7afac7-systemd-units\") pod \"ovnkube-node-942xh\" (UID: \"31ff16f0-6054-4eba-942a-b07a5a7afac7\") " pod="openshift-ovn-kubernetes/ovnkube-node-942xh" Jul 07 15:41:02.838478 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.747880 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0c5071d6-cead-4992-a7e9-9a752b955cae-kubelet-config\") pod \"global-pull-secret-syncer-6qzkn\" (UID: \"0c5071d6-cead-4992-a7e9-9a752b955cae\") " pod="kube-system/global-pull-secret-syncer-6qzkn" Jul 07 15:41:02.838478 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.747880 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/57e72845-ffcd-4aa2-91c8-05d2f00b6b49-multus-socket-dir-parent\") pod \"multus-jzprp\" (UID: \"57e72845-ffcd-4aa2-91c8-05d2f00b6b49\") " pod="openshift-multus/multus-jzprp" Jul 07 15:41:02.839001 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.747881 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d6c50a3b-0739-431c-80b0-8f42c4b00405-hosts-file\") pod \"node-resolver-qggz6\" (UID: \"d6c50a3b-0739-431c-80b0-8f42c4b00405\") " pod="openshift-dns/node-resolver-qggz6" Jul 07 15:41:02.839001 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.747926 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/57e72845-ffcd-4aa2-91c8-05d2f00b6b49-os-release\") pod \"multus-jzprp\" (UID: \"57e72845-ffcd-4aa2-91c8-05d2f00b6b49\") " pod="openshift-multus/multus-jzprp" Jul 07 15:41:02.839001 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.747931 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/76d2d5c2-57b5-4c2d-81f9-73f49cde5fed-device-dir\") pod \"aws-ebs-csi-driver-node-s8x25\" (UID: \"76d2d5c2-57b5-4c2d-81f9-73f49cde5fed\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8x25" Jul 07 15:41:02.839001 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.747951 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d3ca04a5-3622-461f-9b02-0220e1c17814-cnibin\") pod \"multus-additional-cni-plugins-f7x65\" (UID: \"d3ca04a5-3622-461f-9b02-0220e1c17814\") " pod="openshift-multus/multus-additional-cni-plugins-f7x65" Jul 07 15:41:02.839001 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.747973 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d6c50a3b-0739-431c-80b0-8f42c4b00405-hosts-file\") pod \"node-resolver-qggz6\" (UID: \"d6c50a3b-0739-431c-80b0-8f42c4b00405\") " pod="openshift-dns/node-resolver-qggz6" Jul 07 15:41:02.839001 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.747988 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/31ff16f0-6054-4eba-942a-b07a5a7afac7-host-run-netns\") pod \"ovnkube-node-942xh\" (UID: \"31ff16f0-6054-4eba-942a-b07a5a7afac7\") " pod="openshift-ovn-kubernetes/ovnkube-node-942xh" Jul 07 15:41:02.839001 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.748008 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/57e72845-ffcd-4aa2-91c8-05d2f00b6b49-etc-kubernetes\") pod \"multus-jzprp\" (UID: \"57e72845-ffcd-4aa2-91c8-05d2f00b6b49\") " pod="openshift-multus/multus-jzprp" Jul 07 15:41:02.839001 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.748007 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/57e72845-ffcd-4aa2-91c8-05d2f00b6b49-os-release\") pod \"multus-jzprp\" (UID: \"57e72845-ffcd-4aa2-91c8-05d2f00b6b49\") " pod="openshift-multus/multus-jzprp" Jul 07 15:41:02.839001 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.748017 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d3ca04a5-3622-461f-9b02-0220e1c17814-cnibin\") pod \"multus-additional-cni-plugins-f7x65\" (UID: \"d3ca04a5-3622-461f-9b02-0220e1c17814\") " pod="openshift-multus/multus-additional-cni-plugins-f7x65" Jul 07 15:41:02.839001 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.748055 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/76d2d5c2-57b5-4c2d-81f9-73f49cde5fed-socket-dir\") pod \"aws-ebs-csi-driver-node-s8x25\" (UID: \"76d2d5c2-57b5-4c2d-81f9-73f49cde5fed\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8x25" Jul 07 15:41:02.839001 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.748083 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/57e72845-ffcd-4aa2-91c8-05d2f00b6b49-etc-kubernetes\") pod \"multus-jzprp\" (UID: \"57e72845-ffcd-4aa2-91c8-05d2f00b6b49\") " pod="openshift-multus/multus-jzprp" Jul 07 15:41:02.839001 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.748077 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/31ff16f0-6054-4eba-942a-b07a5a7afac7-host-run-netns\") pod \"ovnkube-node-942xh\" (UID: \"31ff16f0-6054-4eba-942a-b07a5a7afac7\") " pod="openshift-ovn-kubernetes/ovnkube-node-942xh" Jul 07 15:41:02.839001 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.748099 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d3ca04a5-3622-461f-9b02-0220e1c17814-cni-binary-copy\") pod \"multus-additional-cni-plugins-f7x65\" (UID: \"d3ca04a5-3622-461f-9b02-0220e1c17814\") " pod="openshift-multus/multus-additional-cni-plugins-f7x65" Jul 07 15:41:02.839001 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.748136 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kmppq\" (UniqueName: \"kubernetes.io/projected/301c1027-169c-47f5-a3ea-89cde78790d2-kube-api-access-kmppq\") pod \"network-metrics-daemon-sdzxq\" (UID: \"301c1027-169c-47f5-a3ea-89cde78790d2\") " pod="openshift-multus/network-metrics-daemon-sdzxq" Jul 07 15:41:02.839001 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.748181 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4e14c00d-b605-4eee-aaa1-8b07158de8f0-etc-sysctl-d\") pod \"tuned-4glxt\" (UID: \"4e14c00d-b605-4eee-aaa1-8b07158de8f0\") " pod="openshift-cluster-node-tuning-operator/tuned-4glxt" Jul 07 15:41:02.839736 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.748190 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/76d2d5c2-57b5-4c2d-81f9-73f49cde5fed-socket-dir\") pod \"aws-ebs-csi-driver-node-s8x25\" (UID: \"76d2d5c2-57b5-4c2d-81f9-73f49cde5fed\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8x25" Jul 07 15:41:02.839736 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.748193 2565 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 07 15:41:02.839736 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.748204 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0c5071d6-cead-4992-a7e9-9a752b955cae-original-pull-secret\") pod \"global-pull-secret-syncer-6qzkn\" (UID: \"0c5071d6-cead-4992-a7e9-9a752b955cae\") " pod="kube-system/global-pull-secret-syncer-6qzkn" Jul 07 15:41:02.839736 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.748293 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/57e72845-ffcd-4aa2-91c8-05d2f00b6b49-host-var-lib-cni-multus\") pod \"multus-jzprp\" (UID: \"57e72845-ffcd-4aa2-91c8-05d2f00b6b49\") " pod="openshift-multus/multus-jzprp" Jul 07 15:41:02.839736 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.748338 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/57e72845-ffcd-4aa2-91c8-05d2f00b6b49-multus-daemon-config\") pod \"multus-jzprp\" (UID: \"57e72845-ffcd-4aa2-91c8-05d2f00b6b49\") " pod="openshift-multus/multus-jzprp" Jul 07 15:41:02.839736 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.748373 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4e14c00d-b605-4eee-aaa1-8b07158de8f0-etc-sysctl-d\") pod \"tuned-4glxt\" (UID: \"4e14c00d-b605-4eee-aaa1-8b07158de8f0\") " pod="openshift-cluster-node-tuning-operator/tuned-4glxt" Jul 07 15:41:02.839736 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.748403 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/57e72845-ffcd-4aa2-91c8-05d2f00b6b49-host-var-lib-cni-multus\") pod \"multus-jzprp\" (UID: \"57e72845-ffcd-4aa2-91c8-05d2f00b6b49\") " pod="openshift-multus/multus-jzprp" Jul 07 15:41:02.839736 ip-10-0-132-65 kubenswrapper[2565]: E0707 15:41:02.748428 2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 07 15:41:02.839736 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.748543 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/76d2d5c2-57b5-4c2d-81f9-73f49cde5fed-sys-fs\") pod \"aws-ebs-csi-driver-node-s8x25\" (UID: \"76d2d5c2-57b5-4c2d-81f9-73f49cde5fed\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8x25" Jul 07 15:41:02.839736 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.748596 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/76d2d5c2-57b5-4c2d-81f9-73f49cde5fed-kubelet-dir\") pod \"aws-ebs-csi-driver-node-s8x25\" (UID: \"76d2d5c2-57b5-4c2d-81f9-73f49cde5fed\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8x25" Jul 07 15:41:02.839736 ip-10-0-132-65 kubenswrapper[2565]: E0707 15:41:02.748605 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0c5071d6-cead-4992-a7e9-9a752b955cae-original-pull-secret podName:0c5071d6-cead-4992-a7e9-9a752b955cae nodeName:}" failed. No retries permitted until 2026-07-07 15:41:03.248580382 +0000 UTC m=+2.205157832 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0c5071d6-cead-4992-a7e9-9a752b955cae-original-pull-secret") pod "global-pull-secret-syncer-6qzkn" (UID: "0c5071d6-cead-4992-a7e9-9a752b955cae") : object "kube-system"/"original-pull-secret" not registered Jul 07 15:41:02.839736 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.748611 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/76d2d5c2-57b5-4c2d-81f9-73f49cde5fed-sys-fs\") pod \"aws-ebs-csi-driver-node-s8x25\" (UID: \"76d2d5c2-57b5-4c2d-81f9-73f49cde5fed\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8x25" Jul 07 15:41:02.839736 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.748606 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d3ca04a5-3622-461f-9b02-0220e1c17814-cni-binary-copy\") pod \"multus-additional-cni-plugins-f7x65\" (UID: \"d3ca04a5-3622-461f-9b02-0220e1c17814\") " pod="openshift-multus/multus-additional-cni-plugins-f7x65" Jul 07 15:41:02.839736 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.748639 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/d3ca04a5-3622-461f-9b02-0220e1c17814-os-release\") pod \"multus-additional-cni-plugins-f7x65\" (UID: \"d3ca04a5-3622-461f-9b02-0220e1c17814\") " pod="openshift-multus/multus-additional-cni-plugins-f7x65" Jul 07 15:41:02.840288 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.748645 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/76d2d5c2-57b5-4c2d-81f9-73f49cde5fed-kubelet-dir\") pod \"aws-ebs-csi-driver-node-s8x25\" (UID: \"76d2d5c2-57b5-4c2d-81f9-73f49cde5fed\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8x25" Jul 07 15:41:02.840288 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.748663 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/31ff16f0-6054-4eba-942a-b07a5a7afac7-host-slash\") pod \"ovnkube-node-942xh\" (UID: \"31ff16f0-6054-4eba-942a-b07a5a7afac7\") " pod="openshift-ovn-kubernetes/ovnkube-node-942xh" Jul 07 15:41:02.840288 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.748679 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/31ff16f0-6054-4eba-942a-b07a5a7afac7-node-log\") pod \"ovnkube-node-942xh\" (UID: \"31ff16f0-6054-4eba-942a-b07a5a7afac7\") " pod="openshift-ovn-kubernetes/ovnkube-node-942xh" Jul 07 15:41:02.840288 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.748694 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/31ff16f0-6054-4eba-942a-b07a5a7afac7-env-overrides\") pod \"ovnkube-node-942xh\" (UID: \"31ff16f0-6054-4eba-942a-b07a5a7afac7\") " pod="openshift-ovn-kubernetes/ovnkube-node-942xh" Jul 07 15:41:02.840288 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.748701 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/31ff16f0-6054-4eba-942a-b07a5a7afac7-host-slash\") pod \"ovnkube-node-942xh\" (UID: \"31ff16f0-6054-4eba-942a-b07a5a7afac7\") " pod="openshift-ovn-kubernetes/ovnkube-node-942xh" Jul 07 15:41:02.840288 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.748715 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/57e72845-ffcd-4aa2-91c8-05d2f00b6b49-cni-binary-copy\") pod \"multus-jzprp\" (UID: \"57e72845-ffcd-4aa2-91c8-05d2f00b6b49\") " pod="openshift-multus/multus-jzprp" Jul 07 15:41:02.840288 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.748725 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/31ff16f0-6054-4eba-942a-b07a5a7afac7-node-log\") pod \"ovnkube-node-942xh\" (UID: \"31ff16f0-6054-4eba-942a-b07a5a7afac7\") " pod="openshift-ovn-kubernetes/ovnkube-node-942xh" Jul 07 15:41:02.840288 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.748749 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/d3ca04a5-3622-461f-9b02-0220e1c17814-os-release\") pod \"multus-additional-cni-plugins-f7x65\" (UID: \"d3ca04a5-3622-461f-9b02-0220e1c17814\") " pod="openshift-multus/multus-additional-cni-plugins-f7x65" Jul 07 15:41:02.840288 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.748757 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4e14c00d-b605-4eee-aaa1-8b07158de8f0-var-lib-kubelet\") pod \"tuned-4glxt\" (UID: \"4e14c00d-b605-4eee-aaa1-8b07158de8f0\") " pod="openshift-cluster-node-tuning-operator/tuned-4glxt" Jul 07 15:41:02.840288 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.748783 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4e14c00d-b605-4eee-aaa1-8b07158de8f0-tmp\") pod \"tuned-4glxt\" (UID: \"4e14c00d-b605-4eee-aaa1-8b07158de8f0\") " pod="openshift-cluster-node-tuning-operator/tuned-4glxt" Jul 07 15:41:02.840288 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.748808 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e5c3e573-02fc-4e63-b475-65ee8b00f34b-serviceca\") pod \"node-ca-9fmf8\" (UID: \"e5c3e573-02fc-4e63-b475-65ee8b00f34b\") " pod="openshift-image-registry/node-ca-9fmf8" Jul 07 15:41:02.840288 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.748837 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/31ff16f0-6054-4eba-942a-b07a5a7afac7-host-kubelet\") pod \"ovnkube-node-942xh\" (UID: \"31ff16f0-6054-4eba-942a-b07a5a7afac7\") " pod="openshift-ovn-kubernetes/ovnkube-node-942xh" Jul 07 15:41:02.840288 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.748808 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4e14c00d-b605-4eee-aaa1-8b07158de8f0-var-lib-kubelet\") pod \"tuned-4glxt\" (UID: \"4e14c00d-b605-4eee-aaa1-8b07158de8f0\") " pod="openshift-cluster-node-tuning-operator/tuned-4glxt" Jul 07 15:41:02.840288 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.748861 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/31ff16f0-6054-4eba-942a-b07a5a7afac7-run-systemd\") pod \"ovnkube-node-942xh\" (UID: \"31ff16f0-6054-4eba-942a-b07a5a7afac7\") " pod="openshift-ovn-kubernetes/ovnkube-node-942xh" Jul 07 15:41:02.840288 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.748901 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/31ff16f0-6054-4eba-942a-b07a5a7afac7-run-systemd\") pod \"ovnkube-node-942xh\" (UID: \"31ff16f0-6054-4eba-942a-b07a5a7afac7\") " pod="openshift-ovn-kubernetes/ovnkube-node-942xh" Jul 07 15:41:02.841483 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.748908 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/31ff16f0-6054-4eba-942a-b07a5a7afac7-ovnkube-config\") pod \"ovnkube-node-942xh\" (UID: \"31ff16f0-6054-4eba-942a-b07a5a7afac7\") " pod="openshift-ovn-kubernetes/ovnkube-node-942xh" Jul 07 15:41:02.841483 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.749161 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/57e72845-ffcd-4aa2-91c8-05d2f00b6b49-host-var-lib-kubelet\") pod \"multus-jzprp\" (UID: \"57e72845-ffcd-4aa2-91c8-05d2f00b6b49\") " pod="openshift-multus/multus-jzprp" Jul 07 15:41:02.841483 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.749225 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e6634bed-1f16-4667-ae9e-c08984716ce5-agent-certs\") pod \"konnectivity-agent-m4wtp\" (UID: \"e6634bed-1f16-4667-ae9e-c08984716ce5\") " pod="kube-system/konnectivity-agent-m4wtp" Jul 07 15:41:02.841483 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.749256 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/4e14c00d-b605-4eee-aaa1-8b07158de8f0-etc-tuned\") pod \"tuned-4glxt\" (UID: \"4e14c00d-b605-4eee-aaa1-8b07158de8f0\") " pod="openshift-cluster-node-tuning-operator/tuned-4glxt" Jul 07 15:41:02.841483 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.749303 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/31ff16f0-6054-4eba-942a-b07a5a7afac7-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-942xh\" (UID: \"31ff16f0-6054-4eba-942a-b07a5a7afac7\") " pod="openshift-ovn-kubernetes/ovnkube-node-942xh" Jul 07 15:41:02.841483 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.749374 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/57e72845-ffcd-4aa2-91c8-05d2f00b6b49-multus-daemon-config\") pod \"multus-jzprp\" (UID: \"57e72845-ffcd-4aa2-91c8-05d2f00b6b49\") " pod="openshift-multus/multus-jzprp" Jul 07 15:41:02.841483 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.749399 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/31ff16f0-6054-4eba-942a-b07a5a7afac7-env-overrides\") pod \"ovnkube-node-942xh\" (UID: \"31ff16f0-6054-4eba-942a-b07a5a7afac7\") " pod="openshift-ovn-kubernetes/ovnkube-node-942xh" Jul 07 15:41:02.841483 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.749380 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m25h7\" (UniqueName: \"kubernetes.io/projected/57e72845-ffcd-4aa2-91c8-05d2f00b6b49-kube-api-access-m25h7\") pod \"multus-jzprp\" (UID: \"57e72845-ffcd-4aa2-91c8-05d2f00b6b49\") " pod="openshift-multus/multus-jzprp" Jul 07 15:41:02.841483 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.749461 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ec51f9ec-f9f7-4390-9954-eb4ab30b8c4d-iptables-alerter-script\") pod \"iptables-alerter-lrpv9\" (UID: \"ec51f9ec-f9f7-4390-9954-eb4ab30b8c4d\") " pod="openshift-network-operator/iptables-alerter-lrpv9" Jul 07 15:41:02.841483 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.749563 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjxct\" (UniqueName: \"kubernetes.io/projected/ec51f9ec-f9f7-4390-9954-eb4ab30b8c4d-kube-api-access-zjxct\") pod \"iptables-alerter-lrpv9\" (UID: \"ec51f9ec-f9f7-4390-9954-eb4ab30b8c4d\") " pod="openshift-network-operator/iptables-alerter-lrpv9" Jul 07 15:41:02.841483 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.749633 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/57e72845-ffcd-4aa2-91c8-05d2f00b6b49-cni-binary-copy\") pod \"multus-jzprp\" (UID: \"57e72845-ffcd-4aa2-91c8-05d2f00b6b49\") " pod="openshift-multus/multus-jzprp" Jul 07 15:41:02.841483 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.749722 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/57e72845-ffcd-4aa2-91c8-05d2f00b6b49-host-var-lib-kubelet\") pod \"multus-jzprp\" (UID: \"57e72845-ffcd-4aa2-91c8-05d2f00b6b49\") " pod="openshift-multus/multus-jzprp" Jul 07 15:41:02.841483 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.749913 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e5c3e573-02fc-4e63-b475-65ee8b00f34b-serviceca\") pod \"node-ca-9fmf8\" (UID: \"e5c3e573-02fc-4e63-b475-65ee8b00f34b\") " pod="openshift-image-registry/node-ca-9fmf8" Jul 07 15:41:02.841483 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:02.750007 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fc4f78a01c7da970c3b1d6baab931c6.slice/crio-d3ec906ec5cad58d5a421f1604059a60a8ae249f84670666241a49750a00c381 WatchSource:0}: Error finding container d3ec906ec5cad58d5a421f1604059a60a8ae249f84670666241a49750a00c381: Status 404 returned error can't find the container with id d3ec906ec5cad58d5a421f1604059a60a8ae249f84670666241a49750a00c381 Jul 07 15:41:02.842121 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.750191 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/31ff16f0-6054-4eba-942a-b07a5a7afac7-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-942xh\" (UID: \"31ff16f0-6054-4eba-942a-b07a5a7afac7\") " pod="openshift-ovn-kubernetes/ovnkube-node-942xh" Jul 07 15:41:02.842121 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.750413 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ec51f9ec-f9f7-4390-9954-eb4ab30b8c4d-iptables-alerter-script\") pod \"iptables-alerter-lrpv9\" (UID: \"ec51f9ec-f9f7-4390-9954-eb4ab30b8c4d\") " pod="openshift-network-operator/iptables-alerter-lrpv9" Jul 07 15:41:02.842121 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.750481 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/31ff16f0-6054-4eba-942a-b07a5a7afac7-host-kubelet\") pod \"ovnkube-node-942xh\" (UID: \"31ff16f0-6054-4eba-942a-b07a5a7afac7\") " pod="openshift-ovn-kubernetes/ovnkube-node-942xh" Jul 07 15:41:02.842121 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.750521 2565 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 07 15:41:02.842121 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.751201 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/31ff16f0-6054-4eba-942a-b07a5a7afac7-ovnkube-config\") pod \"ovnkube-node-942xh\" (UID: \"31ff16f0-6054-4eba-942a-b07a5a7afac7\") " pod="openshift-ovn-kubernetes/ovnkube-node-942xh" Jul 07 15:41:02.842121 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.752666 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4e14c00d-b605-4eee-aaa1-8b07158de8f0-tmp\") pod \"tuned-4glxt\" (UID: \"4e14c00d-b605-4eee-aaa1-8b07158de8f0\") " pod="openshift-cluster-node-tuning-operator/tuned-4glxt" Jul 07 15:41:02.842121 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.753390 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/31ff16f0-6054-4eba-942a-b07a5a7afac7-ovn-node-metrics-cert\") pod \"ovnkube-node-942xh\" (UID: \"31ff16f0-6054-4eba-942a-b07a5a7afac7\") " pod="openshift-ovn-kubernetes/ovnkube-node-942xh" Jul 07 15:41:02.842121 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.753637 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/4e14c00d-b605-4eee-aaa1-8b07158de8f0-etc-tuned\") pod \"tuned-4glxt\" (UID: \"4e14c00d-b605-4eee-aaa1-8b07158de8f0\") " pod="openshift-cluster-node-tuning-operator/tuned-4glxt" Jul 07 15:41:02.842121 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.760877 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e6634bed-1f16-4667-ae9e-c08984716ce5-agent-certs\") pod \"konnectivity-agent-m4wtp\" (UID: \"e6634bed-1f16-4667-ae9e-c08984716ce5\") " pod="kube-system/konnectivity-agent-m4wtp" Jul 07 15:41:02.842121 ip-10-0-132-65 kubenswrapper[2565]: E0707 15:41:02.761154 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 07 15:41:02.842121 ip-10-0-132-65 kubenswrapper[2565]: E0707 15:41:02.761199 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 07 15:41:02.842121 ip-10-0-132-65 kubenswrapper[2565]: E0707 15:41:02.761214 2565 projected.go:196] Error preparing data for projected volume kube-api-access-n6brw for pod openshift-network-diagnostics/network-check-target-94vlx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 15:41:02.842121 ip-10-0-132-65 kubenswrapper[2565]: E0707 15:41:02.761404 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e1f2bb91-99ac-4353-b790-622cb6d40663-kube-api-access-n6brw podName:e1f2bb91-99ac-4353-b790-622cb6d40663 nodeName:}" failed. No retries permitted until 2026-07-07 15:41:03.261377902 +0000 UTC m=+2.217955363 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-n6brw" (UniqueName: "kubernetes.io/projected/e1f2bb91-99ac-4353-b790-622cb6d40663-kube-api-access-n6brw") pod "network-check-target-94vlx" (UID: "e1f2bb91-99ac-4353-b790-622cb6d40663") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 15:41:02.842121 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.763017 2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 07 15:41:02.842121 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.763942 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9l86g\" (UniqueName: \"kubernetes.io/projected/d6c50a3b-0739-431c-80b0-8f42c4b00405-kube-api-access-9l86g\") pod \"node-resolver-qggz6\" (UID: \"d6c50a3b-0739-431c-80b0-8f42c4b00405\") " pod="openshift-dns/node-resolver-qggz6" Jul 07 15:41:02.844818 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.764741 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzqjv\" (UniqueName: \"kubernetes.io/projected/31ff16f0-6054-4eba-942a-b07a5a7afac7-kube-api-access-lzqjv\") pod \"ovnkube-node-942xh\" (UID: \"31ff16f0-6054-4eba-942a-b07a5a7afac7\") " pod="openshift-ovn-kubernetes/ovnkube-node-942xh" Jul 07 15:41:02.844818 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.764808 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxxw4\" (UniqueName: \"kubernetes.io/projected/e5c3e573-02fc-4e63-b475-65ee8b00f34b-kube-api-access-nxxw4\") pod \"node-ca-9fmf8\" (UID: \"e5c3e573-02fc-4e63-b475-65ee8b00f34b\") " pod="openshift-image-registry/node-ca-9fmf8" Jul 07 15:41:02.844818 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.764987 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kmppq\" (UniqueName: \"kubernetes.io/projected/301c1027-169c-47f5-a3ea-89cde78790d2-kube-api-access-kmppq\") pod \"network-metrics-daemon-sdzxq\" (UID: \"301c1027-169c-47f5-a3ea-89cde78790d2\") " pod="openshift-multus/network-metrics-daemon-sdzxq" Jul 07 15:41:02.844818 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.765326 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5dsj\" (UniqueName: \"kubernetes.io/projected/d3ca04a5-3622-461f-9b02-0220e1c17814-kube-api-access-x5dsj\") pod \"multus-additional-cni-plugins-f7x65\" (UID: \"d3ca04a5-3622-461f-9b02-0220e1c17814\") " pod="openshift-multus/multus-additional-cni-plugins-f7x65" Jul 07 15:41:02.844818 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.765639 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdz5m\" (UniqueName: \"kubernetes.io/projected/76d2d5c2-57b5-4c2d-81f9-73f49cde5fed-kube-api-access-tdz5m\") pod \"aws-ebs-csi-driver-node-s8x25\" (UID: \"76d2d5c2-57b5-4c2d-81f9-73f49cde5fed\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8x25" Jul 07 15:41:02.844818 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.765961 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m25h7\" (UniqueName: \"kubernetes.io/projected/57e72845-ffcd-4aa2-91c8-05d2f00b6b49-kube-api-access-m25h7\") pod \"multus-jzprp\" (UID: \"57e72845-ffcd-4aa2-91c8-05d2f00b6b49\") " pod="openshift-multus/multus-jzprp" Jul 07 15:41:02.844818 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.770463 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjxct\" (UniqueName: \"kubernetes.io/projected/ec51f9ec-f9f7-4390-9954-eb4ab30b8c4d-kube-api-access-zjxct\") pod \"iptables-alerter-lrpv9\" (UID: \"ec51f9ec-f9f7-4390-9954-eb4ab30b8c4d\") " pod="openshift-network-operator/iptables-alerter-lrpv9" Jul 07 15:41:02.844818 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.787109 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rk5r\" (UniqueName: \"kubernetes.io/projected/4e14c00d-b605-4eee-aaa1-8b07158de8f0-kube-api-access-7rk5r\") pod \"tuned-4glxt\" (UID: \"4e14c00d-b605-4eee-aaa1-8b07158de8f0\") " pod="openshift-cluster-node-tuning-operator/tuned-4glxt" Jul 07 15:41:02.844818 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.839319 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-jzprp" Jul 07 15:41:02.849119 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:02.847836 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57e72845_ffcd_4aa2_91c8_05d2f00b6b49.slice/crio-41ef25df9526248226e94e8f97b937d144057f3d260858ff4581c4ac4df9be24 WatchSource:0}: Error finding container 41ef25df9526248226e94e8f97b937d144057f3d260858ff4581c4ac4df9be24: Status 404 returned error can't find the container with id 41ef25df9526248226e94e8f97b937d144057f3d260858ff4581c4ac4df9be24 Jul 07 15:41:02.929647 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.867586 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-m4wtp" Jul 07 15:41:02.929647 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:02.876569 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6634bed_1f16_4667_ae9e_c08984716ce5.slice/crio-48eb4d8ccfa01e99480f79d2b65dbf3deff4dc0be24b4e9c30df282499a5b886 WatchSource:0}: Error finding container 48eb4d8ccfa01e99480f79d2b65dbf3deff4dc0be24b4e9c30df282499a5b886: Status 404 returned error can't find the container with id 48eb4d8ccfa01e99480f79d2b65dbf3deff4dc0be24b4e9c30df282499a5b886 Jul 07 15:41:02.929647 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.894787 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-lrpv9" Jul 07 15:41:02.929647 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:02.901906 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec51f9ec_f9f7_4390_9954_eb4ab30b8c4d.slice/crio-5da5b0db8bbf84661c4cfacfc6ebfb30d90caf84d1b183f4e9849d991250c8b2 WatchSource:0}: Error finding container 5da5b0db8bbf84661c4cfacfc6ebfb30d90caf84d1b183f4e9849d991250c8b2: Status 404 returned error can't find the container with id 5da5b0db8bbf84661c4cfacfc6ebfb30d90caf84d1b183f4e9849d991250c8b2 Jul 07 15:41:02.929647 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.911694 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-f7x65" Jul 07 15:41:02.929647 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:02.919172 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3ca04a5_3622_461f_9b02_0220e1c17814.slice/crio-cebcfcbdcff652c3462438d8dde25161944b36b78e630bf1335bec43b09a724f WatchSource:0}: Error finding container cebcfcbdcff652c3462438d8dde25161944b36b78e630bf1335bec43b09a724f: Status 404 returned error can't find the container with id cebcfcbdcff652c3462438d8dde25161944b36b78e630bf1335bec43b09a724f Jul 07 15:41:03.013169 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.944809 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-4glxt" Jul 07 15:41:03.013169 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.950378 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-9fmf8" Jul 07 15:41:03.013169 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:02.952576 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e14c00d_b605_4eee_aaa1_8b07158de8f0.slice/crio-57436bead433daa1bc45fc29806ecaacd60b6b0ed740e0e71cde4127e542eff1 WatchSource:0}: Error finding container 57436bead433daa1bc45fc29806ecaacd60b6b0ed740e0e71cde4127e542eff1: Status 404 returned error can't find the container with id 57436bead433daa1bc45fc29806ecaacd60b6b0ed740e0e71cde4127e542eff1 Jul 07 15:41:03.013169 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:02.971921 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-942xh" Jul 07 15:41:03.013169 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:02.981478 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31ff16f0_6054_4eba_942a_b07a5a7afac7.slice/crio-874f25c1dff136be835dadcc101519d08e4fe860f7ec356ce1a2113b442b9352 WatchSource:0}: Error finding container 874f25c1dff136be835dadcc101519d08e4fe860f7ec356ce1a2113b442b9352: Status 404 returned error can't find the container with id 874f25c1dff136be835dadcc101519d08e4fe860f7ec356ce1a2113b442b9352 Jul 07 15:41:03.013169 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:03.003552 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8x25" Jul 07 15:41:03.013169 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:03.008629 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-qggz6" Jul 07 15:41:03.013169 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:03.011716 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76d2d5c2_57b5_4c2d_81f9_73f49cde5fed.slice/crio-d0c0b9689226bc60ea22020fc019289619e22553382b7e4d685b5e0b148c42ec WatchSource:0}: Error finding container d0c0b9689226bc60ea22020fc019289619e22553382b7e4d685b5e0b148c42ec: Status 404 returned error can't find the container with id d0c0b9689226bc60ea22020fc019289619e22553382b7e4d685b5e0b148c42ec Jul 07 15:41:03.017048 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:03.015900 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6c50a3b_0739_431c_80b0_8f42c4b00405.slice/crio-6162e1f3d7dd02acd2e978c938146d3a9673e76d5be2c2c503b6268991113e0d WatchSource:0}: Error finding container 6162e1f3d7dd02acd2e978c938146d3a9673e76d5be2c2c503b6268991113e0d: Status 404 returned error can't find the container with id 6162e1f3d7dd02acd2e978c938146d3a9673e76d5be2c2c503b6268991113e0d Jul 07 15:41:03.089093 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:03.032834 2565 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 07 15:41:03.253087 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:03.253046 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0c5071d6-cead-4992-a7e9-9a752b955cae-original-pull-secret\") pod \"global-pull-secret-syncer-6qzkn\" (UID: \"0c5071d6-cead-4992-a7e9-9a752b955cae\") " pod="kube-system/global-pull-secret-syncer-6qzkn" Jul 07 15:41:03.253274 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:03.253111 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/301c1027-169c-47f5-a3ea-89cde78790d2-metrics-certs\") pod \"network-metrics-daemon-sdzxq\" (UID: \"301c1027-169c-47f5-a3ea-89cde78790d2\") " pod="openshift-multus/network-metrics-daemon-sdzxq" Jul 07 15:41:03.253274 ip-10-0-132-65 kubenswrapper[2565]: E0707 15:41:03.253214 2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 07 15:41:03.253274 ip-10-0-132-65 kubenswrapper[2565]: E0707 15:41:03.253225 2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 15:41:03.253518 ip-10-0-132-65 kubenswrapper[2565]: E0707 15:41:03.253292 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0c5071d6-cead-4992-a7e9-9a752b955cae-original-pull-secret podName:0c5071d6-cead-4992-a7e9-9a752b955cae nodeName:}" failed. No retries permitted until 2026-07-07 15:41:04.253270115 +0000 UTC m=+3.209847567 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0c5071d6-cead-4992-a7e9-9a752b955cae-original-pull-secret") pod "global-pull-secret-syncer-6qzkn" (UID: "0c5071d6-cead-4992-a7e9-9a752b955cae") : object "kube-system"/"original-pull-secret" not registered Jul 07 15:41:03.253518 ip-10-0-132-65 kubenswrapper[2565]: E0707 15:41:03.253311 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/301c1027-169c-47f5-a3ea-89cde78790d2-metrics-certs podName:301c1027-169c-47f5-a3ea-89cde78790d2 nodeName:}" failed. No retries permitted until 2026-07-07 15:41:04.25330254 +0000 UTC m=+3.209879990 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/301c1027-169c-47f5-a3ea-89cde78790d2-metrics-certs") pod "network-metrics-daemon-sdzxq" (UID: "301c1027-169c-47f5-a3ea-89cde78790d2") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 15:41:03.354486 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:03.354409 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6brw\" (UniqueName: \"kubernetes.io/projected/e1f2bb91-99ac-4353-b790-622cb6d40663-kube-api-access-n6brw\") pod \"network-check-target-94vlx\" (UID: \"e1f2bb91-99ac-4353-b790-622cb6d40663\") " pod="openshift-network-diagnostics/network-check-target-94vlx" Jul 07 15:41:03.354657 ip-10-0-132-65 kubenswrapper[2565]: E0707 15:41:03.354579 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 07 15:41:03.354657 ip-10-0-132-65 kubenswrapper[2565]: E0707 15:41:03.354605 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 07 15:41:03.354657 ip-10-0-132-65 kubenswrapper[2565]: E0707 15:41:03.354621 2565 projected.go:196] Error preparing data for projected volume kube-api-access-n6brw for pod openshift-network-diagnostics/network-check-target-94vlx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 15:41:03.354824 ip-10-0-132-65 kubenswrapper[2565]: E0707 15:41:03.354695 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e1f2bb91-99ac-4353-b790-622cb6d40663-kube-api-access-n6brw podName:e1f2bb91-99ac-4353-b790-622cb6d40663 nodeName:}" failed. No retries permitted until 2026-07-07 15:41:04.354668686 +0000 UTC m=+3.311246151 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-n6brw" (UniqueName: "kubernetes.io/projected/e1f2bb91-99ac-4353-b790-622cb6d40663-kube-api-access-n6brw") pod "network-check-target-94vlx" (UID: "e1f2bb91-99ac-4353-b790-622cb6d40663") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 15:41:03.395424 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:03.395210 2565 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 07 15:41:03.578139 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:03.577592 2565 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-06 15:36:02 +0000 UTC" deadline="2028-01-01 20:47:01.370726568 +0000 UTC" Jul 07 15:41:03.578139 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:03.577628 2565 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13037h5m57.793103571s" Jul 07 15:41:03.659137 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:03.658107 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sdzxq" Jul 07 15:41:03.659137 ip-10-0-132-65 kubenswrapper[2565]: E0707 15:41:03.658260 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sdzxq" podUID="301c1027-169c-47f5-a3ea-89cde78790d2" Jul 07 15:41:03.659137 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:03.658615 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-94vlx" Jul 07 15:41:03.659137 ip-10-0-132-65 kubenswrapper[2565]: E0707 15:41:03.658734 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-94vlx" podUID="e1f2bb91-99ac-4353-b790-622cb6d40663" Jul 07 15:41:03.659137 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:03.658800 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6qzkn" Jul 07 15:41:03.659137 ip-10-0-132-65 kubenswrapper[2565]: E0707 15:41:03.658917 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6qzkn" podUID="0c5071d6-cead-4992-a7e9-9a752b955cae" Jul 07 15:41:03.668861 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:03.668760 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8x25" event={"ID":"76d2d5c2-57b5-4c2d-81f9-73f49cde5fed","Type":"ContainerStarted","Data":"d0c0b9689226bc60ea22020fc019289619e22553382b7e4d685b5e0b148c42ec"} Jul 07 15:41:03.682114 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:03.682065 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-942xh" event={"ID":"31ff16f0-6054-4eba-942a-b07a5a7afac7","Type":"ContainerStarted","Data":"874f25c1dff136be835dadcc101519d08e4fe860f7ec356ce1a2113b442b9352"} Jul 07 15:41:03.690528 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:03.690472 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-9fmf8" event={"ID":"e5c3e573-02fc-4e63-b475-65ee8b00f34b","Type":"ContainerStarted","Data":"51058bca04b70ce9d6ac7751f1f4dbb61eda8dc836ecfff4203c09028bc1375f"} Jul 07 15:41:03.696041 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:03.695907 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-4glxt" event={"ID":"4e14c00d-b605-4eee-aaa1-8b07158de8f0","Type":"ContainerStarted","Data":"57436bead433daa1bc45fc29806ecaacd60b6b0ed740e0e71cde4127e542eff1"} Jul 07 15:41:03.712125 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:03.712012 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-f7x65" event={"ID":"d3ca04a5-3622-461f-9b02-0220e1c17814","Type":"ContainerStarted","Data":"cebcfcbdcff652c3462438d8dde25161944b36b78e630bf1335bec43b09a724f"} Jul 07 15:41:03.725631 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:03.725541 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-m4wtp" event={"ID":"e6634bed-1f16-4667-ae9e-c08984716ce5","Type":"ContainerStarted","Data":"48eb4d8ccfa01e99480f79d2b65dbf3deff4dc0be24b4e9c30df282499a5b886"} Jul 07 15:41:03.749167 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:03.749081 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-65.ec2.internal" event={"ID":"98ae1d05f22dc745f3b265fc9d76c5f3","Type":"ContainerStarted","Data":"cc78e3905fe2f40334ea86fb502840830a3b6ab6ecd31bd6e188394a244c0619"} Jul 07 15:41:03.770840 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:03.770802 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-qggz6" event={"ID":"d6c50a3b-0739-431c-80b0-8f42c4b00405","Type":"ContainerStarted","Data":"6162e1f3d7dd02acd2e978c938146d3a9673e76d5be2c2c503b6268991113e0d"} Jul 07 15:41:03.777273 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:03.777237 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-lrpv9" event={"ID":"ec51f9ec-f9f7-4390-9954-eb4ab30b8c4d","Type":"ContainerStarted","Data":"5da5b0db8bbf84661c4cfacfc6ebfb30d90caf84d1b183f4e9849d991250c8b2"} Jul 07 15:41:03.787773 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:03.787697 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-jzprp" event={"ID":"57e72845-ffcd-4aa2-91c8-05d2f00b6b49","Type":"ContainerStarted","Data":"41ef25df9526248226e94e8f97b937d144057f3d260858ff4581c4ac4df9be24"} Jul 07 15:41:03.796273 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:03.796240 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-65.ec2.internal" event={"ID":"9fc4f78a01c7da970c3b1d6baab931c6","Type":"ContainerStarted","Data":"d3ec906ec5cad58d5a421f1604059a60a8ae249f84670666241a49750a00c381"} Jul 07 15:41:04.090376 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:04.090295 2565 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 07 15:41:04.265112 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:04.265070 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0c5071d6-cead-4992-a7e9-9a752b955cae-original-pull-secret\") pod \"global-pull-secret-syncer-6qzkn\" (UID: \"0c5071d6-cead-4992-a7e9-9a752b955cae\") " pod="kube-system/global-pull-secret-syncer-6qzkn" Jul 07 15:41:04.265291 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:04.265144 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/301c1027-169c-47f5-a3ea-89cde78790d2-metrics-certs\") pod \"network-metrics-daemon-sdzxq\" (UID: \"301c1027-169c-47f5-a3ea-89cde78790d2\") " pod="openshift-multus/network-metrics-daemon-sdzxq" Jul 07 15:41:04.265361 ip-10-0-132-65 kubenswrapper[2565]: E0707 15:41:04.265299 2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 15:41:04.265418 ip-10-0-132-65 kubenswrapper[2565]: E0707 15:41:04.265363 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/301c1027-169c-47f5-a3ea-89cde78790d2-metrics-certs podName:301c1027-169c-47f5-a3ea-89cde78790d2 nodeName:}" failed. No retries permitted until 2026-07-07 15:41:06.265344623 +0000 UTC m=+5.221922075 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/301c1027-169c-47f5-a3ea-89cde78790d2-metrics-certs") pod "network-metrics-daemon-sdzxq" (UID: "301c1027-169c-47f5-a3ea-89cde78790d2") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 15:41:04.265829 ip-10-0-132-65 kubenswrapper[2565]: E0707 15:41:04.265811 2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 07 15:41:04.265908 ip-10-0-132-65 kubenswrapper[2565]: E0707 15:41:04.265866 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0c5071d6-cead-4992-a7e9-9a752b955cae-original-pull-secret podName:0c5071d6-cead-4992-a7e9-9a752b955cae nodeName:}" failed. No retries permitted until 2026-07-07 15:41:06.265850307 +0000 UTC m=+5.222427759 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0c5071d6-cead-4992-a7e9-9a752b955cae-original-pull-secret") pod "global-pull-secret-syncer-6qzkn" (UID: "0c5071d6-cead-4992-a7e9-9a752b955cae") : object "kube-system"/"original-pull-secret" not registered Jul 07 15:41:04.365637 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:04.365551 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6brw\" (UniqueName: \"kubernetes.io/projected/e1f2bb91-99ac-4353-b790-622cb6d40663-kube-api-access-n6brw\") pod \"network-check-target-94vlx\" (UID: \"e1f2bb91-99ac-4353-b790-622cb6d40663\") " pod="openshift-network-diagnostics/network-check-target-94vlx" Jul 07 15:41:04.365795 ip-10-0-132-65 kubenswrapper[2565]: E0707 15:41:04.365723 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 07 15:41:04.365795 ip-10-0-132-65 kubenswrapper[2565]: E0707 15:41:04.365742 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 07 15:41:04.365795 ip-10-0-132-65 kubenswrapper[2565]: E0707 15:41:04.365754 2565 projected.go:196] Error preparing data for projected volume kube-api-access-n6brw for pod openshift-network-diagnostics/network-check-target-94vlx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 15:41:04.365955 ip-10-0-132-65 kubenswrapper[2565]: E0707 15:41:04.365813 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e1f2bb91-99ac-4353-b790-622cb6d40663-kube-api-access-n6brw podName:e1f2bb91-99ac-4353-b790-622cb6d40663 nodeName:}" failed. No retries permitted until 2026-07-07 15:41:06.365794742 +0000 UTC m=+5.322372188 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-n6brw" (UniqueName: "kubernetes.io/projected/e1f2bb91-99ac-4353-b790-622cb6d40663-kube-api-access-n6brw") pod "network-check-target-94vlx" (UID: "e1f2bb91-99ac-4353-b790-622cb6d40663") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 15:41:04.578499 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:04.578419 2565 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-06 15:36:02 +0000 UTC" deadline="2028-02-08 09:04:20.600830146 +0000 UTC" Jul 07 15:41:04.578499 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:04.578481 2565 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13937h23m16.022353136s" Jul 07 15:41:05.656242 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:05.655740 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-94vlx" Jul 07 15:41:05.656242 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:05.655827 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6qzkn" Jul 07 15:41:05.656242 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:05.655777 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sdzxq" Jul 07 15:41:05.656242 ip-10-0-132-65 kubenswrapper[2565]: E0707 15:41:05.655911 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-94vlx" podUID="e1f2bb91-99ac-4353-b790-622cb6d40663" Jul 07 15:41:05.656242 ip-10-0-132-65 kubenswrapper[2565]: E0707 15:41:05.656040 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sdzxq" podUID="301c1027-169c-47f5-a3ea-89cde78790d2" Jul 07 15:41:05.656242 ip-10-0-132-65 kubenswrapper[2565]: E0707 15:41:05.656113 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6qzkn" podUID="0c5071d6-cead-4992-a7e9-9a752b955cae" Jul 07 15:41:06.282793 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:06.282743 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0c5071d6-cead-4992-a7e9-9a752b955cae-original-pull-secret\") pod \"global-pull-secret-syncer-6qzkn\" (UID: \"0c5071d6-cead-4992-a7e9-9a752b955cae\") " pod="kube-system/global-pull-secret-syncer-6qzkn" Jul 07 15:41:06.282947 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:06.282821 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/301c1027-169c-47f5-a3ea-89cde78790d2-metrics-certs\") pod \"network-metrics-daemon-sdzxq\" (UID: \"301c1027-169c-47f5-a3ea-89cde78790d2\") " pod="openshift-multus/network-metrics-daemon-sdzxq" Jul 07 15:41:06.283005 ip-10-0-132-65 kubenswrapper[2565]: E0707 15:41:06.282968 2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 15:41:06.283056 ip-10-0-132-65 kubenswrapper[2565]: E0707 15:41:06.283030 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/301c1027-169c-47f5-a3ea-89cde78790d2-metrics-certs podName:301c1027-169c-47f5-a3ea-89cde78790d2 nodeName:}" failed. No retries permitted until 2026-07-07 15:41:10.283011706 +0000 UTC m=+9.239589152 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/301c1027-169c-47f5-a3ea-89cde78790d2-metrics-certs") pod "network-metrics-daemon-sdzxq" (UID: "301c1027-169c-47f5-a3ea-89cde78790d2") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 15:41:06.283573 ip-10-0-132-65 kubenswrapper[2565]: E0707 15:41:06.283406 2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 07 15:41:06.283573 ip-10-0-132-65 kubenswrapper[2565]: E0707 15:41:06.283461 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0c5071d6-cead-4992-a7e9-9a752b955cae-original-pull-secret podName:0c5071d6-cead-4992-a7e9-9a752b955cae nodeName:}" failed. No retries permitted until 2026-07-07 15:41:10.283440965 +0000 UTC m=+9.240018414 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0c5071d6-cead-4992-a7e9-9a752b955cae-original-pull-secret") pod "global-pull-secret-syncer-6qzkn" (UID: "0c5071d6-cead-4992-a7e9-9a752b955cae") : object "kube-system"/"original-pull-secret" not registered Jul 07 15:41:06.384349 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:06.384106 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6brw\" (UniqueName: \"kubernetes.io/projected/e1f2bb91-99ac-4353-b790-622cb6d40663-kube-api-access-n6brw\") pod \"network-check-target-94vlx\" (UID: \"e1f2bb91-99ac-4353-b790-622cb6d40663\") " pod="openshift-network-diagnostics/network-check-target-94vlx" Jul 07 15:41:06.384349 ip-10-0-132-65 kubenswrapper[2565]: E0707 15:41:06.384262 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 07 15:41:06.384349 ip-10-0-132-65 kubenswrapper[2565]: E0707 15:41:06.384281 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 07 15:41:06.384349 ip-10-0-132-65 kubenswrapper[2565]: E0707 15:41:06.384296 2565 projected.go:196] Error preparing data for projected volume kube-api-access-n6brw for pod openshift-network-diagnostics/network-check-target-94vlx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 15:41:06.384349 ip-10-0-132-65 kubenswrapper[2565]: E0707 15:41:06.384360 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e1f2bb91-99ac-4353-b790-622cb6d40663-kube-api-access-n6brw podName:e1f2bb91-99ac-4353-b790-622cb6d40663 nodeName:}" failed. No retries permitted until 2026-07-07 15:41:10.384342802 +0000 UTC m=+9.340920248 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-n6brw" (UniqueName: "kubernetes.io/projected/e1f2bb91-99ac-4353-b790-622cb6d40663-kube-api-access-n6brw") pod "network-check-target-94vlx" (UID: "e1f2bb91-99ac-4353-b790-622cb6d40663") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 15:41:07.655143 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:07.655107 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-94vlx" Jul 07 15:41:07.655627 ip-10-0-132-65 kubenswrapper[2565]: E0707 15:41:07.655244 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-94vlx" podUID="e1f2bb91-99ac-4353-b790-622cb6d40663" Jul 07 15:41:07.655627 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:07.655616 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sdzxq" Jul 07 15:41:07.655731 ip-10-0-132-65 kubenswrapper[2565]: E0707 15:41:07.655688 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sdzxq" podUID="301c1027-169c-47f5-a3ea-89cde78790d2" Jul 07 15:41:07.655807 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:07.655793 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6qzkn" Jul 07 15:41:07.655859 ip-10-0-132-65 kubenswrapper[2565]: E0707 15:41:07.655850 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6qzkn" podUID="0c5071d6-cead-4992-a7e9-9a752b955cae" Jul 07 15:41:09.655076 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:09.655035 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sdzxq" Jul 07 15:41:09.655438 ip-10-0-132-65 kubenswrapper[2565]: E0707 15:41:09.655177 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sdzxq" podUID="301c1027-169c-47f5-a3ea-89cde78790d2" Jul 07 15:41:09.655438 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:09.655276 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6qzkn" Jul 07 15:41:09.655438 ip-10-0-132-65 kubenswrapper[2565]: E0707 15:41:09.655378 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6qzkn" podUID="0c5071d6-cead-4992-a7e9-9a752b955cae" Jul 07 15:41:09.655592 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:09.655472 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-94vlx" Jul 07 15:41:09.655592 ip-10-0-132-65 kubenswrapper[2565]: E0707 15:41:09.655569 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-94vlx" podUID="e1f2bb91-99ac-4353-b790-622cb6d40663" Jul 07 15:41:10.318689 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:10.318647 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0c5071d6-cead-4992-a7e9-9a752b955cae-original-pull-secret\") pod \"global-pull-secret-syncer-6qzkn\" (UID: \"0c5071d6-cead-4992-a7e9-9a752b955cae\") " pod="kube-system/global-pull-secret-syncer-6qzkn" Jul 07 15:41:10.318858 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:10.318716 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/301c1027-169c-47f5-a3ea-89cde78790d2-metrics-certs\") pod \"network-metrics-daemon-sdzxq\" (UID: \"301c1027-169c-47f5-a3ea-89cde78790d2\") " pod="openshift-multus/network-metrics-daemon-sdzxq" Jul 07 15:41:10.318858 ip-10-0-132-65 kubenswrapper[2565]: E0707 15:41:10.318821 2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 07 15:41:10.318999 ip-10-0-132-65 kubenswrapper[2565]: E0707 15:41:10.318892 2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 15:41:10.318999 ip-10-0-132-65 kubenswrapper[2565]: E0707 15:41:10.318898 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0c5071d6-cead-4992-a7e9-9a752b955cae-original-pull-secret podName:0c5071d6-cead-4992-a7e9-9a752b955cae nodeName:}" failed. No retries permitted until 2026-07-07 15:41:18.318876534 +0000 UTC m=+17.275454023 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0c5071d6-cead-4992-a7e9-9a752b955cae-original-pull-secret") pod "global-pull-secret-syncer-6qzkn" (UID: "0c5071d6-cead-4992-a7e9-9a752b955cae") : object "kube-system"/"original-pull-secret" not registered Jul 07 15:41:10.318999 ip-10-0-132-65 kubenswrapper[2565]: E0707 15:41:10.318962 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/301c1027-169c-47f5-a3ea-89cde78790d2-metrics-certs podName:301c1027-169c-47f5-a3ea-89cde78790d2 nodeName:}" failed. No retries permitted until 2026-07-07 15:41:18.318944183 +0000 UTC m=+17.275521634 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/301c1027-169c-47f5-a3ea-89cde78790d2-metrics-certs") pod "network-metrics-daemon-sdzxq" (UID: "301c1027-169c-47f5-a3ea-89cde78790d2") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 15:41:10.419816 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:10.419779 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6brw\" (UniqueName: \"kubernetes.io/projected/e1f2bb91-99ac-4353-b790-622cb6d40663-kube-api-access-n6brw\") pod \"network-check-target-94vlx\" (UID: \"e1f2bb91-99ac-4353-b790-622cb6d40663\") " pod="openshift-network-diagnostics/network-check-target-94vlx" Jul 07 15:41:10.420006 ip-10-0-132-65 kubenswrapper[2565]: E0707 15:41:10.419946 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 07 15:41:10.420006 ip-10-0-132-65 kubenswrapper[2565]: E0707 15:41:10.419967 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 07 15:41:10.420006 ip-10-0-132-65 kubenswrapper[2565]: E0707 15:41:10.419980 2565 projected.go:196] Error preparing data for projected volume kube-api-access-n6brw for pod openshift-network-diagnostics/network-check-target-94vlx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 15:41:10.420169 ip-10-0-132-65 kubenswrapper[2565]: E0707 15:41:10.420060 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e1f2bb91-99ac-4353-b790-622cb6d40663-kube-api-access-n6brw podName:e1f2bb91-99ac-4353-b790-622cb6d40663 nodeName:}" failed. No retries permitted until 2026-07-07 15:41:18.420041727 +0000 UTC m=+17.376619176 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-n6brw" (UniqueName: "kubernetes.io/projected/e1f2bb91-99ac-4353-b790-622cb6d40663-kube-api-access-n6brw") pod "network-check-target-94vlx" (UID: "e1f2bb91-99ac-4353-b790-622cb6d40663") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 15:41:11.656766 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:11.656180 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6qzkn" Jul 07 15:41:11.656766 ip-10-0-132-65 kubenswrapper[2565]: E0707 15:41:11.656296 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6qzkn" podUID="0c5071d6-cead-4992-a7e9-9a752b955cae" Jul 07 15:41:11.656766 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:11.656353 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sdzxq" Jul 07 15:41:11.656766 ip-10-0-132-65 kubenswrapper[2565]: E0707 15:41:11.656430 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sdzxq" podUID="301c1027-169c-47f5-a3ea-89cde78790d2" Jul 07 15:41:11.656766 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:11.656561 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-94vlx" Jul 07 15:41:11.656766 ip-10-0-132-65 kubenswrapper[2565]: E0707 15:41:11.656686 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-94vlx" podUID="e1f2bb91-99ac-4353-b790-622cb6d40663" Jul 07 15:41:13.654992 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:13.654953 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6qzkn" Jul 07 15:41:13.655450 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:13.654954 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sdzxq" Jul 07 15:41:13.655450 ip-10-0-132-65 kubenswrapper[2565]: E0707 15:41:13.655087 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6qzkn" podUID="0c5071d6-cead-4992-a7e9-9a752b955cae" Jul 07 15:41:13.655450 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:13.655093 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-94vlx" Jul 07 15:41:13.655450 ip-10-0-132-65 kubenswrapper[2565]: E0707 15:41:13.655211 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sdzxq" podUID="301c1027-169c-47f5-a3ea-89cde78790d2" Jul 07 15:41:13.655450 ip-10-0-132-65 kubenswrapper[2565]: E0707 15:41:13.655273 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-94vlx" podUID="e1f2bb91-99ac-4353-b790-622cb6d40663" Jul 07 15:41:15.654617 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:15.654575 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6qzkn" Jul 07 15:41:15.655073 ip-10-0-132-65 kubenswrapper[2565]: E0707 15:41:15.654729 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6qzkn" podUID="0c5071d6-cead-4992-a7e9-9a752b955cae" Jul 07 15:41:15.655073 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:15.654575 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-94vlx" Jul 07 15:41:15.655073 ip-10-0-132-65 kubenswrapper[2565]: E0707 15:41:15.654818 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-94vlx" podUID="e1f2bb91-99ac-4353-b790-622cb6d40663" Jul 07 15:41:15.655073 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:15.654578 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sdzxq" Jul 07 15:41:15.655073 ip-10-0-132-65 kubenswrapper[2565]: E0707 15:41:15.654906 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sdzxq" podUID="301c1027-169c-47f5-a3ea-89cde78790d2" Jul 07 15:41:17.655138 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:17.655098 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-94vlx" Jul 07 15:41:17.655616 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:17.655105 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sdzxq" Jul 07 15:41:17.655616 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:17.655228 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6qzkn" Jul 07 15:41:17.655616 ip-10-0-132-65 kubenswrapper[2565]: E0707 15:41:17.655231 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-94vlx" podUID="e1f2bb91-99ac-4353-b790-622cb6d40663" Jul 07 15:41:17.655616 ip-10-0-132-65 kubenswrapper[2565]: E0707 15:41:17.655311 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sdzxq" podUID="301c1027-169c-47f5-a3ea-89cde78790d2" Jul 07 15:41:17.655616 ip-10-0-132-65 kubenswrapper[2565]: E0707 15:41:17.655416 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6qzkn" podUID="0c5071d6-cead-4992-a7e9-9a752b955cae" Jul 07 15:41:18.383982 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:18.383948 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/301c1027-169c-47f5-a3ea-89cde78790d2-metrics-certs\") pod \"network-metrics-daemon-sdzxq\" (UID: \"301c1027-169c-47f5-a3ea-89cde78790d2\") " pod="openshift-multus/network-metrics-daemon-sdzxq" Jul 07 15:41:18.384178 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:18.384039 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0c5071d6-cead-4992-a7e9-9a752b955cae-original-pull-secret\") pod \"global-pull-secret-syncer-6qzkn\" (UID: \"0c5071d6-cead-4992-a7e9-9a752b955cae\") " pod="kube-system/global-pull-secret-syncer-6qzkn" Jul 07 15:41:18.384178 ip-10-0-132-65 kubenswrapper[2565]: E0707 15:41:18.384116 2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 07 15:41:18.384178 ip-10-0-132-65 kubenswrapper[2565]: E0707 15:41:18.384174 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0c5071d6-cead-4992-a7e9-9a752b955cae-original-pull-secret podName:0c5071d6-cead-4992-a7e9-9a752b955cae nodeName:}" failed. No retries permitted until 2026-07-07 15:41:34.384160869 +0000 UTC m=+33.340738314 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0c5071d6-cead-4992-a7e9-9a752b955cae-original-pull-secret") pod "global-pull-secret-syncer-6qzkn" (UID: "0c5071d6-cead-4992-a7e9-9a752b955cae") : object "kube-system"/"original-pull-secret" not registered Jul 07 15:41:18.384322 ip-10-0-132-65 kubenswrapper[2565]: E0707 15:41:18.384116 2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 15:41:18.384322 ip-10-0-132-65 kubenswrapper[2565]: E0707 15:41:18.384238 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/301c1027-169c-47f5-a3ea-89cde78790d2-metrics-certs podName:301c1027-169c-47f5-a3ea-89cde78790d2 nodeName:}" failed. No retries permitted until 2026-07-07 15:41:34.384226157 +0000 UTC m=+33.340803605 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/301c1027-169c-47f5-a3ea-89cde78790d2-metrics-certs") pod "network-metrics-daemon-sdzxq" (UID: "301c1027-169c-47f5-a3ea-89cde78790d2") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 15:41:18.485331 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:18.485285 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6brw\" (UniqueName: \"kubernetes.io/projected/e1f2bb91-99ac-4353-b790-622cb6d40663-kube-api-access-n6brw\") pod \"network-check-target-94vlx\" (UID: \"e1f2bb91-99ac-4353-b790-622cb6d40663\") " pod="openshift-network-diagnostics/network-check-target-94vlx" Jul 07 15:41:18.485539 ip-10-0-132-65 kubenswrapper[2565]: E0707 15:41:18.485452 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 07 15:41:18.485539 ip-10-0-132-65 kubenswrapper[2565]: E0707 15:41:18.485473 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 07 15:41:18.485539 ip-10-0-132-65 kubenswrapper[2565]: E0707 15:41:18.485486 2565 projected.go:196] Error preparing data for projected volume kube-api-access-n6brw for pod openshift-network-diagnostics/network-check-target-94vlx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 15:41:18.485695 ip-10-0-132-65 kubenswrapper[2565]: E0707 15:41:18.485564 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e1f2bb91-99ac-4353-b790-622cb6d40663-kube-api-access-n6brw podName:e1f2bb91-99ac-4353-b790-622cb6d40663 nodeName:}" failed. No retries permitted until 2026-07-07 15:41:34.485547449 +0000 UTC m=+33.442124918 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-n6brw" (UniqueName: "kubernetes.io/projected/e1f2bb91-99ac-4353-b790-622cb6d40663-kube-api-access-n6brw") pod "network-check-target-94vlx" (UID: "e1f2bb91-99ac-4353-b790-622cb6d40663") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 15:41:19.654978 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:19.654938 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sdzxq" Jul 07 15:41:19.655447 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:19.654938 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-94vlx" Jul 07 15:41:19.655447 ip-10-0-132-65 kubenswrapper[2565]: E0707 15:41:19.655086 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sdzxq" podUID="301c1027-169c-47f5-a3ea-89cde78790d2" Jul 07 15:41:19.655447 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:19.654959 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6qzkn" Jul 07 15:41:19.655447 ip-10-0-132-65 kubenswrapper[2565]: E0707 15:41:19.655147 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-94vlx" podUID="e1f2bb91-99ac-4353-b790-622cb6d40663" Jul 07 15:41:19.655447 ip-10-0-132-65 kubenswrapper[2565]: E0707 15:41:19.655205 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6qzkn" podUID="0c5071d6-cead-4992-a7e9-9a752b955cae" Jul 07 15:41:21.655872 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:21.655641 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sdzxq" Jul 07 15:41:21.656290 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:21.655704 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-94vlx" Jul 07 15:41:21.656290 ip-10-0-132-65 kubenswrapper[2565]: E0707 15:41:21.655965 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sdzxq" podUID="301c1027-169c-47f5-a3ea-89cde78790d2" Jul 07 15:41:21.656290 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:21.655732 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6qzkn" Jul 07 15:41:21.656290 ip-10-0-132-65 kubenswrapper[2565]: E0707 15:41:21.656072 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-94vlx" podUID="e1f2bb91-99ac-4353-b790-622cb6d40663" Jul 07 15:41:21.656290 ip-10-0-132-65 kubenswrapper[2565]: E0707 15:41:21.656183 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6qzkn" podUID="0c5071d6-cead-4992-a7e9-9a752b955cae" Jul 07 15:41:23.655002 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:23.654811 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6qzkn" Jul 07 15:41:23.655599 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:23.654820 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-94vlx" Jul 07 15:41:23.655599 ip-10-0-132-65 kubenswrapper[2565]: E0707 15:41:23.655108 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6qzkn" podUID="0c5071d6-cead-4992-a7e9-9a752b955cae" Jul 07 15:41:23.655599 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:23.654876 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sdzxq" Jul 07 15:41:23.655599 ip-10-0-132-65 kubenswrapper[2565]: E0707 15:41:23.655162 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-94vlx" podUID="e1f2bb91-99ac-4353-b790-622cb6d40663" Jul 07 15:41:23.655599 ip-10-0-132-65 kubenswrapper[2565]: E0707 15:41:23.655228 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sdzxq" podUID="301c1027-169c-47f5-a3ea-89cde78790d2" Jul 07 15:41:23.847931 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:23.847854 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-qggz6" event={"ID":"d6c50a3b-0739-431c-80b0-8f42c4b00405","Type":"ContainerStarted","Data":"28204a1498177120f1e74e19c18fbb942b3eec5f195761401f49d25a264572aa"} Jul 07 15:41:23.849516 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:23.849468 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-jzprp" event={"ID":"57e72845-ffcd-4aa2-91c8-05d2f00b6b49","Type":"ContainerStarted","Data":"fe1154e87bdaa5df5e4bb07ae64556da9203e0e27fc3f98deba6ffe859648f4f"} Jul 07 15:41:23.850976 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:23.850952 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-65.ec2.internal" event={"ID":"9fc4f78a01c7da970c3b1d6baab931c6","Type":"ContainerStarted","Data":"2515222c9f62fe7a3984027b631df99830d25527ae862f00bbe95a2fd762093e"} Jul 07 15:41:23.852618 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:23.852594 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8x25" event={"ID":"76d2d5c2-57b5-4c2d-81f9-73f49cde5fed","Type":"ContainerStarted","Data":"643a4d1525a4746c49d56369b2eefaa0c17b0fd4463297cdc88c2686a23fa7bd"} Jul 07 15:41:23.855852 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:23.855784 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-942xh_31ff16f0-6054-4eba-942a-b07a5a7afac7/ovn-acl-logging/0.log" Jul 07 15:41:23.856126 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:23.856106 2565 generic.go:356] "Generic (PLEG): container finished" podID="31ff16f0-6054-4eba-942a-b07a5a7afac7" containerID="ede0a95a3efdf71101d65e93f7ce1e7d3d68afa1a115e58f4f1c944365118d9a" exitCode=1 Jul 07 15:41:23.856228 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:23.856166 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-942xh" event={"ID":"31ff16f0-6054-4eba-942a-b07a5a7afac7","Type":"ContainerStarted","Data":"d82f39228c6667f5f6f14faffd345f8ba50b750bfc0b85f9f0d4731fc30d330f"} Jul 07 15:41:23.856296 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:23.856247 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-942xh" event={"ID":"31ff16f0-6054-4eba-942a-b07a5a7afac7","Type":"ContainerStarted","Data":"c488a1c3f8e88416c5138baf46ecc42f8a739b792c558bfe174b3453c960855b"} Jul 07 15:41:23.856343 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:23.856300 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-942xh" event={"ID":"31ff16f0-6054-4eba-942a-b07a5a7afac7","Type":"ContainerStarted","Data":"0f7be49b2443d55f42e9d5253f861d9c2f7a6e3f0697710e25928d658333416c"} Jul 07 15:41:23.856395 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:23.856349 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-942xh" event={"ID":"31ff16f0-6054-4eba-942a-b07a5a7afac7","Type":"ContainerStarted","Data":"25f04ec55996fd5cd1b9e6030653992d1c17ff7bb8834c01d99b4a1cc3034886"} Jul 07 15:41:23.856446 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:23.856402 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-942xh" event={"ID":"31ff16f0-6054-4eba-942a-b07a5a7afac7","Type":"ContainerDied","Data":"ede0a95a3efdf71101d65e93f7ce1e7d3d68afa1a115e58f4f1c944365118d9a"} Jul 07 15:41:23.856515 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:23.856454 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-942xh" event={"ID":"31ff16f0-6054-4eba-942a-b07a5a7afac7","Type":"ContainerStarted","Data":"62604075b7157eab6e7c6480137c9320919508282337cf715d02d81bfaee03ab"} Jul 07 15:41:23.857604 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:23.857580 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-9fmf8" event={"ID":"e5c3e573-02fc-4e63-b475-65ee8b00f34b","Type":"ContainerStarted","Data":"b096fc60a6fc565a2989aa7176e07e4f97ad2a5eec506b1d37331bda98c3deae"} Jul 07 15:41:23.859056 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:23.859032 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-4glxt" event={"ID":"4e14c00d-b605-4eee-aaa1-8b07158de8f0","Type":"ContainerStarted","Data":"9c59eb10dee9ab46220847ba5781ab66bbbdff63259a2a8c00c80b52edeb01a3"} Jul 07 15:41:23.860485 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:23.860454 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-f7x65" event={"ID":"d3ca04a5-3622-461f-9b02-0220e1c17814","Type":"ContainerStarted","Data":"679e0bab8576d44825128ca7496c532cedbad6676c93eddca22f507e6b58f7e2"} Jul 07 15:41:23.861848 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:23.861826 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-m4wtp" event={"ID":"e6634bed-1f16-4667-ae9e-c08984716ce5","Type":"ContainerStarted","Data":"b11be44c99f70d2be51bd07be4a11ee6cf5e082afa27c99f992928fe61ec0234"} Jul 07 15:41:23.863584 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:23.863530 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-65.ec2.internal" event={"ID":"98ae1d05f22dc745f3b265fc9d76c5f3","Type":"ContainerStarted","Data":"235643c7c7742d924842464fd2796ad977dedd183ce2d4991d444d578f37735a"} Jul 07 15:41:23.867161 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:23.867124 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-qggz6" podStartSLOduration=3.185302709 podStartE2EDuration="22.867113125s" totalImagesPullingTime="19.681810416s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 15:41:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:41:23.866899339 +0000 UTC m=+22.823476807" watchObservedRunningTime="2026-07-07 15:41:23.867113125 +0000 UTC m=+22.823690591" Jul 07 15:41:23.883322 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:23.883286 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-9fmf8" podStartSLOduration=3.144443279 podStartE2EDuration="22.883272292s" totalImagesPullingTime="19.738829013s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 15:41:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:41:23.883086644 +0000 UTC m=+22.839664110" watchObservedRunningTime="2026-07-07 15:41:23.883272292 +0000 UTC m=+22.839849761" Jul 07 15:41:23.947789 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:23.947745 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-65.ec2.internal" podStartSLOduration=21.947729882 podStartE2EDuration="21.947729882s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 15:41:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:41:23.946923282 +0000 UTC m=+22.903500749" watchObservedRunningTime="2026-07-07 15:41:23.947729882 +0000 UTC m=+22.904307349" Jul 07 15:41:23.967073 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:23.967030 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-jzprp" podStartSLOduration=3.0824803 podStartE2EDuration="22.967016919s" totalImagesPullingTime="19.884536619s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 15:41:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:41:23.966461462 +0000 UTC m=+22.923038932" watchObservedRunningTime="2026-07-07 15:41:23.967016919 +0000 UTC m=+22.923594420" Jul 07 15:41:24.002925 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:24.002902 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-m4wtp" Jul 07 15:41:24.003563 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:24.003532 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-m4wtp" Jul 07 15:41:24.003754 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:24.003714 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-m4wtp" podStartSLOduration=3.18265911 podStartE2EDuration="23.003698056s" totalImagesPullingTime="19.821038946s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:41:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:41:23.984449071 +0000 UTC m=+22.941026539" watchObservedRunningTime="2026-07-07 15:41:24.003698056 +0000 UTC m=+22.960275572" Jul 07 15:41:24.003815 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:24.003791 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-4glxt" podStartSLOduration=3.241159823 podStartE2EDuration="23.003784221s" totalImagesPullingTime="19.762624398s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 15:41:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:41:24.003306013 +0000 UTC m=+22.959883481" watchObservedRunningTime="2026-07-07 15:41:24.003784221 +0000 UTC m=+22.960361689" Jul 07 15:41:24.868041 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:24.868008 2565 generic.go:356] "Generic (PLEG): container finished" podID="d3ca04a5-3622-461f-9b02-0220e1c17814" containerID="679e0bab8576d44825128ca7496c532cedbad6676c93eddca22f507e6b58f7e2" exitCode=0 Jul 07 15:41:24.868941 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:24.868101 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-f7x65" event={"ID":"d3ca04a5-3622-461f-9b02-0220e1c17814","Type":"ContainerDied","Data":"679e0bab8576d44825128ca7496c532cedbad6676c93eddca22f507e6b58f7e2"} Jul 07 15:41:24.870405 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:24.870379 2565 generic.go:356] "Generic (PLEG): container finished" podID="98ae1d05f22dc745f3b265fc9d76c5f3" containerID="235643c7c7742d924842464fd2796ad977dedd183ce2d4991d444d578f37735a" exitCode=0 Jul 07 15:41:24.870521 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:24.870468 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-65.ec2.internal" event={"ID":"98ae1d05f22dc745f3b265fc9d76c5f3","Type":"ContainerDied","Data":"235643c7c7742d924842464fd2796ad977dedd183ce2d4991d444d578f37735a"} Jul 07 15:41:24.870581 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:24.870524 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-65.ec2.internal" event={"ID":"98ae1d05f22dc745f3b265fc9d76c5f3","Type":"ContainerStarted","Data":"7a89506e1fce19a4d0079934c8b244ff180a854ba876bc10f70235d0ac5e56ca"} Jul 07 15:41:24.872261 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:24.872234 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-lrpv9" event={"ID":"ec51f9ec-f9f7-4390-9954-eb4ab30b8c4d","Type":"ContainerStarted","Data":"d0bdd79638ba0c85fe96d889ec0c6cbde39522148e0d46c92db8f5ef5c227bcd"} Jul 07 15:41:24.873257 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:24.873232 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-m4wtp" Jul 07 15:41:24.873684 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:24.873664 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-m4wtp" Jul 07 15:41:24.932283 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:24.932239 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-65.ec2.internal" podStartSLOduration=22.932225674 podStartE2EDuration="22.932225674s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 15:41:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:41:24.931913948 +0000 UTC m=+23.888491416" watchObservedRunningTime="2026-07-07 15:41:24.932225674 +0000 UTC m=+23.888803141" Jul 07 15:41:24.951249 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:24.951202 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-lrpv9" podStartSLOduration=4.135367107 podStartE2EDuration="23.951189156s" totalImagesPullingTime="19.815822049s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 15:41:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:41:24.950852273 +0000 UTC m=+23.907429740" watchObservedRunningTime="2026-07-07 15:41:24.951189156 +0000 UTC m=+23.907766623" Jul 07 15:41:25.178796 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:25.178777 2565 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 07 15:41:25.553059 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:25.552886 2565 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-07T15:41:25.178793154Z","UUID":"92f80839-9019-4826-8332-817d1bb11f72","Handler":null,"Name":"","Endpoint":""} Jul 07 15:41:25.554861 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:25.554828 2565 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 07 15:41:25.554991 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:25.554879 2565 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 07 15:41:25.654969 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:25.654933 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-94vlx" Jul 07 15:41:25.654969 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:25.654974 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sdzxq" Jul 07 15:41:25.655227 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:25.654933 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6qzkn" Jul 07 15:41:25.655227 ip-10-0-132-65 kubenswrapper[2565]: E0707 15:41:25.655130 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-94vlx" podUID="e1f2bb91-99ac-4353-b790-622cb6d40663" Jul 07 15:41:25.655227 ip-10-0-132-65 kubenswrapper[2565]: E0707 15:41:25.655169 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6qzkn" podUID="0c5071d6-cead-4992-a7e9-9a752b955cae" Jul 07 15:41:25.655392 ip-10-0-132-65 kubenswrapper[2565]: E0707 15:41:25.655243 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sdzxq" podUID="301c1027-169c-47f5-a3ea-89cde78790d2" Jul 07 15:41:25.876833 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:25.876793 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8x25" event={"ID":"76d2d5c2-57b5-4c2d-81f9-73f49cde5fed","Type":"ContainerStarted","Data":"41156d460ca1dc32b85b5f71a7115d6d344d75e9282d5c76cfbfa5deeed49664"} Jul 07 15:41:25.881245 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:25.881221 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-942xh_31ff16f0-6054-4eba-942a-b07a5a7afac7/ovn-acl-logging/0.log" Jul 07 15:41:25.881787 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:25.881752 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-942xh" event={"ID":"31ff16f0-6054-4eba-942a-b07a5a7afac7","Type":"ContainerStarted","Data":"f79846b20cc4d07b754f8d8227b68af4075a29f9cb3d05eebfc3d0519fefe428"} Jul 07 15:41:26.887249 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:26.887200 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8x25" event={"ID":"76d2d5c2-57b5-4c2d-81f9-73f49cde5fed","Type":"ContainerStarted","Data":"dff05c85955f8bb183f32d449be1015084b317451d40f75e3e718524ee3e5823"} Jul 07 15:41:26.916659 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:26.916607 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s8x25" podStartSLOduration=3.912815112 podStartE2EDuration="25.916592036s" totalImagesPullingTime="22.003776924s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 15:41:01 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:41:26.915981463 +0000 UTC m=+25.872558930" watchObservedRunningTime="2026-07-07 15:41:26.916592036 +0000 UTC m=+25.873169502" Jul 07 15:41:27.654915 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:27.654878 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6qzkn" Jul 07 15:41:27.655099 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:27.654878 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sdzxq" Jul 07 15:41:27.655099 ip-10-0-132-65 kubenswrapper[2565]: E0707 15:41:27.655009 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6qzkn" podUID="0c5071d6-cead-4992-a7e9-9a752b955cae" Jul 07 15:41:27.655213 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:27.654874 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-94vlx" Jul 07 15:41:27.655213 ip-10-0-132-65 kubenswrapper[2565]: E0707 15:41:27.655101 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sdzxq" podUID="301c1027-169c-47f5-a3ea-89cde78790d2" Jul 07 15:41:27.655213 ip-10-0-132-65 kubenswrapper[2565]: E0707 15:41:27.655191 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-94vlx" podUID="e1f2bb91-99ac-4353-b790-622cb6d40663" Jul 07 15:41:29.655072 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:29.654888 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6qzkn" Jul 07 15:41:29.655828 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:29.654890 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-94vlx" Jul 07 15:41:29.655828 ip-10-0-132-65 kubenswrapper[2565]: E0707 15:41:29.655161 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6qzkn" podUID="0c5071d6-cead-4992-a7e9-9a752b955cae" Jul 07 15:41:29.655828 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:29.654953 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sdzxq" Jul 07 15:41:29.655828 ip-10-0-132-65 kubenswrapper[2565]: E0707 15:41:29.655234 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-94vlx" podUID="e1f2bb91-99ac-4353-b790-622cb6d40663" Jul 07 15:41:29.655828 ip-10-0-132-65 kubenswrapper[2565]: E0707 15:41:29.655297 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sdzxq" podUID="301c1027-169c-47f5-a3ea-89cde78790d2" Jul 07 15:41:29.895594 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:29.895557 2565 generic.go:356] "Generic (PLEG): container finished" podID="d3ca04a5-3622-461f-9b02-0220e1c17814" containerID="3e779a2aaf8e9b377e23266ebae3d8c0cb93dd0b5273816e37ef02037704a483" exitCode=0 Jul 07 15:41:29.895766 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:29.895609 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-f7x65" event={"ID":"d3ca04a5-3622-461f-9b02-0220e1c17814","Type":"ContainerDied","Data":"3e779a2aaf8e9b377e23266ebae3d8c0cb93dd0b5273816e37ef02037704a483"} Jul 07 15:41:29.899086 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:29.899070 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-942xh_31ff16f0-6054-4eba-942a-b07a5a7afac7/ovn-acl-logging/0.log" Jul 07 15:41:29.899398 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:29.899381 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-942xh" event={"ID":"31ff16f0-6054-4eba-942a-b07a5a7afac7","Type":"ContainerStarted","Data":"61b05cbe8c4828c57b3e7336e8968baccb4545184fec22df8a03dd64a4bb00f1"} Jul 07 15:41:29.899629 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:29.899608 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-942xh" Jul 07 15:41:29.899731 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:29.899679 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-942xh" Jul 07 15:41:29.899803 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:29.899790 2565 scope.go:117] "RemoveContainer" containerID="ede0a95a3efdf71101d65e93f7ce1e7d3d68afa1a115e58f4f1c944365118d9a" Jul 07 15:41:29.915944 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:29.915919 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-942xh" Jul 07 15:41:30.905730 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:30.905553 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-942xh_31ff16f0-6054-4eba-942a-b07a5a7afac7/ovn-acl-logging/0.log" Jul 07 15:41:30.906245 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:30.906060 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-942xh" event={"ID":"31ff16f0-6054-4eba-942a-b07a5a7afac7","Type":"ContainerStarted","Data":"5f0de384161d157d05235941a26a25e5743f0a5178e8e431620866b23b28102c"} Jul 07 15:41:30.906387 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:30.906366 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-942xh" Jul 07 15:41:30.920988 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:30.920966 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-942xh" Jul 07 15:41:30.957522 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:30.956325 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-942xh" podStartSLOduration=9.835612169000001 podStartE2EDuration="29.956303037s" totalImagesPullingTime="20.120690868s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 15:41:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:41:30.943385418 +0000 UTC m=+29.899962885" watchObservedRunningTime="2026-07-07 15:41:30.956303037 +0000 UTC m=+29.912880505" Jul 07 15:41:31.096137 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:31.096079 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-94vlx"] Jul 07 15:41:31.096307 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:31.096263 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-94vlx" Jul 07 15:41:31.096404 ip-10-0-132-65 kubenswrapper[2565]: E0707 15:41:31.096375 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-94vlx" podUID="e1f2bb91-99ac-4353-b790-622cb6d40663" Jul 07 15:41:31.096788 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:31.096766 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-6qzkn"] Jul 07 15:41:31.096863 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:31.096848 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6qzkn" Jul 07 15:41:31.096931 ip-10-0-132-65 kubenswrapper[2565]: E0707 15:41:31.096914 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6qzkn" podUID="0c5071d6-cead-4992-a7e9-9a752b955cae" Jul 07 15:41:31.103672 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:31.103647 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-sdzxq"] Jul 07 15:41:31.103786 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:31.103748 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sdzxq" Jul 07 15:41:31.103849 ip-10-0-132-65 kubenswrapper[2565]: E0707 15:41:31.103830 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sdzxq" podUID="301c1027-169c-47f5-a3ea-89cde78790d2" Jul 07 15:41:31.910476 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:31.910439 2565 generic.go:356] "Generic (PLEG): container finished" podID="d3ca04a5-3622-461f-9b02-0220e1c17814" containerID="927436ec71681266e5b7bc31d2509356353eab5c87cdf3d64a1f49f2f3d2c645" exitCode=0 Jul 07 15:41:31.910906 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:31.910528 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-f7x65" event={"ID":"d3ca04a5-3622-461f-9b02-0220e1c17814","Type":"ContainerDied","Data":"927436ec71681266e5b7bc31d2509356353eab5c87cdf3d64a1f49f2f3d2c645"} Jul 07 15:41:32.654973 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:32.654896 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-94vlx" Jul 07 15:41:32.655109 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:32.655021 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sdzxq" Jul 07 15:41:32.655109 ip-10-0-132-65 kubenswrapper[2565]: E0707 15:41:32.655032 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-94vlx" podUID="e1f2bb91-99ac-4353-b790-622cb6d40663" Jul 07 15:41:32.655109 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:32.655083 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6qzkn" Jul 07 15:41:32.655324 ip-10-0-132-65 kubenswrapper[2565]: E0707 15:41:32.655190 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6qzkn" podUID="0c5071d6-cead-4992-a7e9-9a752b955cae" Jul 07 15:41:32.655324 ip-10-0-132-65 kubenswrapper[2565]: E0707 15:41:32.655282 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sdzxq" podUID="301c1027-169c-47f5-a3ea-89cde78790d2" Jul 07 15:41:33.918431 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:33.918249 2565 generic.go:356] "Generic (PLEG): container finished" podID="d3ca04a5-3622-461f-9b02-0220e1c17814" containerID="0d561db91f235e36194107dd09dadc45bb0fc23231ba42069010a7832e137548" exitCode=0 Jul 07 15:41:33.918431 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:33.918325 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-f7x65" event={"ID":"d3ca04a5-3622-461f-9b02-0220e1c17814","Type":"ContainerDied","Data":"0d561db91f235e36194107dd09dadc45bb0fc23231ba42069010a7832e137548"} Jul 07 15:41:34.404333 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:34.404304 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0c5071d6-cead-4992-a7e9-9a752b955cae-original-pull-secret\") pod \"global-pull-secret-syncer-6qzkn\" (UID: \"0c5071d6-cead-4992-a7e9-9a752b955cae\") " pod="kube-system/global-pull-secret-syncer-6qzkn" Jul 07 15:41:34.404542 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:34.404346 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/301c1027-169c-47f5-a3ea-89cde78790d2-metrics-certs\") pod \"network-metrics-daemon-sdzxq\" (UID: \"301c1027-169c-47f5-a3ea-89cde78790d2\") " pod="openshift-multus/network-metrics-daemon-sdzxq" Jul 07 15:41:34.404542 ip-10-0-132-65 kubenswrapper[2565]: E0707 15:41:34.404461 2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 15:41:34.404542 ip-10-0-132-65 kubenswrapper[2565]: E0707 15:41:34.404463 2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 07 15:41:34.404542 ip-10-0-132-65 kubenswrapper[2565]: E0707 15:41:34.404535 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/301c1027-169c-47f5-a3ea-89cde78790d2-metrics-certs podName:301c1027-169c-47f5-a3ea-89cde78790d2 nodeName:}" failed. No retries permitted until 2026-07-07 15:42:06.404518631 +0000 UTC m=+65.361096077 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/301c1027-169c-47f5-a3ea-89cde78790d2-metrics-certs") pod "network-metrics-daemon-sdzxq" (UID: "301c1027-169c-47f5-a3ea-89cde78790d2") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 15:41:34.404706 ip-10-0-132-65 kubenswrapper[2565]: E0707 15:41:34.404551 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0c5071d6-cead-4992-a7e9-9a752b955cae-original-pull-secret podName:0c5071d6-cead-4992-a7e9-9a752b955cae nodeName:}" failed. No retries permitted until 2026-07-07 15:42:06.404544543 +0000 UTC m=+65.361121988 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0c5071d6-cead-4992-a7e9-9a752b955cae-original-pull-secret") pod "global-pull-secret-syncer-6qzkn" (UID: "0c5071d6-cead-4992-a7e9-9a752b955cae") : object "kube-system"/"original-pull-secret" not registered Jul 07 15:41:34.504846 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:34.504811 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6brw\" (UniqueName: \"kubernetes.io/projected/e1f2bb91-99ac-4353-b790-622cb6d40663-kube-api-access-n6brw\") pod \"network-check-target-94vlx\" (UID: \"e1f2bb91-99ac-4353-b790-622cb6d40663\") " pod="openshift-network-diagnostics/network-check-target-94vlx" Jul 07 15:41:34.505016 ip-10-0-132-65 kubenswrapper[2565]: E0707 15:41:34.504937 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 07 15:41:34.505016 ip-10-0-132-65 kubenswrapper[2565]: E0707 15:41:34.504959 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 07 15:41:34.505016 ip-10-0-132-65 kubenswrapper[2565]: E0707 15:41:34.504969 2565 projected.go:196] Error preparing data for projected volume kube-api-access-n6brw for pod openshift-network-diagnostics/network-check-target-94vlx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 15:41:34.505118 ip-10-0-132-65 kubenswrapper[2565]: E0707 15:41:34.505019 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e1f2bb91-99ac-4353-b790-622cb6d40663-kube-api-access-n6brw podName:e1f2bb91-99ac-4353-b790-622cb6d40663 nodeName:}" failed. No retries permitted until 2026-07-07 15:42:06.505004123 +0000 UTC m=+65.461581567 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-n6brw" (UniqueName: "kubernetes.io/projected/e1f2bb91-99ac-4353-b790-622cb6d40663-kube-api-access-n6brw") pod "network-check-target-94vlx" (UID: "e1f2bb91-99ac-4353-b790-622cb6d40663") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 15:41:34.654999 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:34.654915 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6qzkn" Jul 07 15:41:34.654999 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:34.654956 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sdzxq" Jul 07 15:41:34.654999 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:34.654952 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-94vlx" Jul 07 15:41:34.655260 ip-10-0-132-65 kubenswrapper[2565]: E0707 15:41:34.655062 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6qzkn" podUID="0c5071d6-cead-4992-a7e9-9a752b955cae" Jul 07 15:41:34.655260 ip-10-0-132-65 kubenswrapper[2565]: E0707 15:41:34.655169 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sdzxq" podUID="301c1027-169c-47f5-a3ea-89cde78790d2" Jul 07 15:41:34.655260 ip-10-0-132-65 kubenswrapper[2565]: E0707 15:41:34.655249 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-94vlx" podUID="e1f2bb91-99ac-4353-b790-622cb6d40663" Jul 07 15:41:35.252504 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:35.252443 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:41:35.361637 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:35.361601 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:41:35.696690 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:35.696652 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:41:35.873201 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:35.873168 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:41:35.982700 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:35.982604 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:41:36.348815 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:36.348781 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:41:36.530583 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:36.530547 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:41:36.654937 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:36.654806 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6qzkn" Jul 07 15:41:36.654937 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:36.654802 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sdzxq" Jul 07 15:41:36.654937 ip-10-0-132-65 kubenswrapper[2565]: E0707 15:41:36.654927 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6qzkn" podUID="0c5071d6-cead-4992-a7e9-9a752b955cae" Jul 07 15:41:36.655202 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:36.654953 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-94vlx" Jul 07 15:41:36.655202 ip-10-0-132-65 kubenswrapper[2565]: E0707 15:41:36.655026 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sdzxq" podUID="301c1027-169c-47f5-a3ea-89cde78790d2" Jul 07 15:41:36.655202 ip-10-0-132-65 kubenswrapper[2565]: E0707 15:41:36.655075 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-94vlx" podUID="e1f2bb91-99ac-4353-b790-622cb6d40663" Jul 07 15:41:36.847629 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:36.847600 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-65.ec2.internal" event="NodeReady" Jul 07 15:41:36.847814 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:36.847725 2565 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 07 15:41:36.899004 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:36.898971 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-qhqsf"] Jul 07 15:41:36.901883 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:36.901864 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-qhqsf" Jul 07 15:41:36.904632 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:36.904605 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 07 15:41:36.904976 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:36.904920 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 07 15:41:36.904976 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:36.904927 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-w75vl\"" Jul 07 15:41:36.906392 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:36.906372 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-t6xpc"] Jul 07 15:41:36.908138 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:36.908122 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-t6xpc" Jul 07 15:41:36.910949 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:36.910930 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 07 15:41:36.911063 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:36.910969 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 07 15:41:36.911135 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:36.911074 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-q5k6l\"" Jul 07 15:41:36.911135 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:36.911115 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 07 15:41:36.913573 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:36.913547 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-qhqsf"] Jul 07 15:41:36.919668 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:36.919577 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-t6xpc"] Jul 07 15:41:37.022324 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:37.022283 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cb4ddf17-3bbe-4d75-9510-99f4a61da167-metrics-tls\") pod \"dns-default-qhqsf\" (UID: \"cb4ddf17-3bbe-4d75-9510-99f4a61da167\") " pod="openshift-dns/dns-default-qhqsf" Jul 07 15:41:37.022527 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:37.022338 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/cb4ddf17-3bbe-4d75-9510-99f4a61da167-config-volume\") pod \"dns-default-qhqsf\" (UID: \"cb4ddf17-3bbe-4d75-9510-99f4a61da167\") " pod="openshift-dns/dns-default-qhqsf" Jul 07 15:41:37.022527 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:37.022365 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zkbnh\" (UniqueName: \"kubernetes.io/projected/cb4ddf17-3bbe-4d75-9510-99f4a61da167-kube-api-access-zkbnh\") pod \"dns-default-qhqsf\" (UID: \"cb4ddf17-3bbe-4d75-9510-99f4a61da167\") " pod="openshift-dns/dns-default-qhqsf" Jul 07 15:41:37.022527 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:37.022506 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nl82v\" (UniqueName: \"kubernetes.io/projected/4097f76e-a65f-4fa3-a14f-491cd0d5ce3f-kube-api-access-nl82v\") pod \"ingress-canary-t6xpc\" (UID: \"4097f76e-a65f-4fa3-a14f-491cd0d5ce3f\") " pod="openshift-ingress-canary/ingress-canary-t6xpc" Jul 07 15:41:37.022657 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:37.022560 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb4ddf17-3bbe-4d75-9510-99f4a61da167-tmp-dir\") pod \"dns-default-qhqsf\" (UID: \"cb4ddf17-3bbe-4d75-9510-99f4a61da167\") " pod="openshift-dns/dns-default-qhqsf" Jul 07 15:41:37.022657 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:37.022588 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4097f76e-a65f-4fa3-a14f-491cd0d5ce3f-cert\") pod \"ingress-canary-t6xpc\" (UID: \"4097f76e-a65f-4fa3-a14f-491cd0d5ce3f\") " pod="openshift-ingress-canary/ingress-canary-t6xpc" Jul 07 15:41:37.123658 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:37.123617 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nl82v\" (UniqueName: \"kubernetes.io/projected/4097f76e-a65f-4fa3-a14f-491cd0d5ce3f-kube-api-access-nl82v\") pod \"ingress-canary-t6xpc\" (UID: \"4097f76e-a65f-4fa3-a14f-491cd0d5ce3f\") " pod="openshift-ingress-canary/ingress-canary-t6xpc" Jul 07 15:41:37.123836 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:37.123669 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb4ddf17-3bbe-4d75-9510-99f4a61da167-tmp-dir\") pod \"dns-default-qhqsf\" (UID: \"cb4ddf17-3bbe-4d75-9510-99f4a61da167\") " pod="openshift-dns/dns-default-qhqsf" Jul 07 15:41:37.123836 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:37.123694 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4097f76e-a65f-4fa3-a14f-491cd0d5ce3f-cert\") pod \"ingress-canary-t6xpc\" (UID: \"4097f76e-a65f-4fa3-a14f-491cd0d5ce3f\") " pod="openshift-ingress-canary/ingress-canary-t6xpc" Jul 07 15:41:37.123836 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:37.123720 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cb4ddf17-3bbe-4d75-9510-99f4a61da167-metrics-tls\") pod \"dns-default-qhqsf\" (UID: \"cb4ddf17-3bbe-4d75-9510-99f4a61da167\") " pod="openshift-dns/dns-default-qhqsf" Jul 07 15:41:37.123836 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:37.123758 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/cb4ddf17-3bbe-4d75-9510-99f4a61da167-config-volume\") pod \"dns-default-qhqsf\" (UID: \"cb4ddf17-3bbe-4d75-9510-99f4a61da167\") " pod="openshift-dns/dns-default-qhqsf" Jul 07 15:41:37.123836 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:37.123780 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkbnh\" (UniqueName: \"kubernetes.io/projected/cb4ddf17-3bbe-4d75-9510-99f4a61da167-kube-api-access-zkbnh\") pod \"dns-default-qhqsf\" (UID: \"cb4ddf17-3bbe-4d75-9510-99f4a61da167\") " pod="openshift-dns/dns-default-qhqsf" Jul 07 15:41:37.124072 ip-10-0-132-65 kubenswrapper[2565]: E0707 15:41:37.123869 2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 07 15:41:37.124072 ip-10-0-132-65 kubenswrapper[2565]: E0707 15:41:37.123939 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb4ddf17-3bbe-4d75-9510-99f4a61da167-metrics-tls podName:cb4ddf17-3bbe-4d75-9510-99f4a61da167 nodeName:}" failed. No retries permitted until 2026-07-07 15:41:37.623917834 +0000 UTC m=+36.580495279 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/cb4ddf17-3bbe-4d75-9510-99f4a61da167-metrics-tls") pod "dns-default-qhqsf" (UID: "cb4ddf17-3bbe-4d75-9510-99f4a61da167") : secret "dns-default-metrics-tls" not found Jul 07 15:41:37.124175 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:37.124144 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb4ddf17-3bbe-4d75-9510-99f4a61da167-tmp-dir\") pod \"dns-default-qhqsf\" (UID: \"cb4ddf17-3bbe-4d75-9510-99f4a61da167\") " pod="openshift-dns/dns-default-qhqsf" Jul 07 15:41:37.124556 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:37.124536 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/cb4ddf17-3bbe-4d75-9510-99f4a61da167-config-volume\") pod \"dns-default-qhqsf\" (UID: \"cb4ddf17-3bbe-4d75-9510-99f4a61da167\") " pod="openshift-dns/dns-default-qhqsf" Jul 07 15:41:37.128173 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:37.128147 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4097f76e-a65f-4fa3-a14f-491cd0d5ce3f-cert\") pod \"ingress-canary-t6xpc\" (UID: \"4097f76e-a65f-4fa3-a14f-491cd0d5ce3f\") " pod="openshift-ingress-canary/ingress-canary-t6xpc" Jul 07 15:41:37.133760 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:37.133740 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zkbnh\" (UniqueName: \"kubernetes.io/projected/cb4ddf17-3bbe-4d75-9510-99f4a61da167-kube-api-access-zkbnh\") pod \"dns-default-qhqsf\" (UID: \"cb4ddf17-3bbe-4d75-9510-99f4a61da167\") " pod="openshift-dns/dns-default-qhqsf" Jul 07 15:41:37.133868 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:37.133782 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nl82v\" (UniqueName: \"kubernetes.io/projected/4097f76e-a65f-4fa3-a14f-491cd0d5ce3f-kube-api-access-nl82v\") pod \"ingress-canary-t6xpc\" (UID: \"4097f76e-a65f-4fa3-a14f-491cd0d5ce3f\") " pod="openshift-ingress-canary/ingress-canary-t6xpc" Jul 07 15:41:37.221055 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:37.220974 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-t6xpc" Jul 07 15:41:37.627458 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:37.627424 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cb4ddf17-3bbe-4d75-9510-99f4a61da167-metrics-tls\") pod \"dns-default-qhqsf\" (UID: \"cb4ddf17-3bbe-4d75-9510-99f4a61da167\") " pod="openshift-dns/dns-default-qhqsf" Jul 07 15:41:37.628115 ip-10-0-132-65 kubenswrapper[2565]: E0707 15:41:37.627617 2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 07 15:41:37.628115 ip-10-0-132-65 kubenswrapper[2565]: E0707 15:41:37.627697 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb4ddf17-3bbe-4d75-9510-99f4a61da167-metrics-tls podName:cb4ddf17-3bbe-4d75-9510-99f4a61da167 nodeName:}" failed. No retries permitted until 2026-07-07 15:41:38.627676687 +0000 UTC m=+37.584254136 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/cb4ddf17-3bbe-4d75-9510-99f4a61da167-metrics-tls") pod "dns-default-qhqsf" (UID: "cb4ddf17-3bbe-4d75-9510-99f4a61da167") : secret "dns-default-metrics-tls" not found Jul 07 15:41:38.324873 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:38.324729 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-t6xpc"] Jul 07 15:41:38.328507 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:38.328468 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4097f76e_a65f_4fa3_a14f_491cd0d5ce3f.slice/crio-431c71d22cad4a9818fd34e9546a3cc4f1f1af00bb0911a8509d1727ee3d50a6 WatchSource:0}: Error finding container 431c71d22cad4a9818fd34e9546a3cc4f1f1af00bb0911a8509d1727ee3d50a6: Status 404 returned error can't find the container with id 431c71d22cad4a9818fd34e9546a3cc4f1f1af00bb0911a8509d1727ee3d50a6 Jul 07 15:41:38.633679 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:38.633648 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cb4ddf17-3bbe-4d75-9510-99f4a61da167-metrics-tls\") pod \"dns-default-qhqsf\" (UID: \"cb4ddf17-3bbe-4d75-9510-99f4a61da167\") " pod="openshift-dns/dns-default-qhqsf" Jul 07 15:41:38.634242 ip-10-0-132-65 kubenswrapper[2565]: E0707 15:41:38.633834 2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 07 15:41:38.634242 ip-10-0-132-65 kubenswrapper[2565]: E0707 15:41:38.633912 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb4ddf17-3bbe-4d75-9510-99f4a61da167-metrics-tls podName:cb4ddf17-3bbe-4d75-9510-99f4a61da167 nodeName:}" failed. No retries permitted until 2026-07-07 15:41:40.633891138 +0000 UTC m=+39.590468585 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/cb4ddf17-3bbe-4d75-9510-99f4a61da167-metrics-tls") pod "dns-default-qhqsf" (UID: "cb4ddf17-3bbe-4d75-9510-99f4a61da167") : secret "dns-default-metrics-tls" not found Jul 07 15:41:38.654952 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:38.654918 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6qzkn" Jul 07 15:41:38.655092 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:38.654970 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-94vlx" Jul 07 15:41:38.655092 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:38.655059 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sdzxq" Jul 07 15:41:38.659157 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:38.659033 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 07 15:41:38.659157 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:38.659037 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-k55hz\"" Jul 07 15:41:38.659157 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:38.659072 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 07 15:41:38.659157 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:38.659037 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-t2q2r\"" Jul 07 15:41:38.659157 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:38.659082 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 07 15:41:38.659157 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:38.659079 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 07 15:41:38.873336 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:38.873252 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:41:38.936217 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:38.936135 2565 generic.go:356] "Generic (PLEG): container finished" podID="d3ca04a5-3622-461f-9b02-0220e1c17814" containerID="fc0c559ec4c047fa634f1f91a07d5fec80610b24bf8c3a6d3005e00146751958" exitCode=0 Jul 07 15:41:38.936364 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:38.936234 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-f7x65" event={"ID":"d3ca04a5-3622-461f-9b02-0220e1c17814","Type":"ContainerDied","Data":"fc0c559ec4c047fa634f1f91a07d5fec80610b24bf8c3a6d3005e00146751958"} Jul 07 15:41:38.937463 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:38.937438 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-t6xpc" event={"ID":"4097f76e-a65f-4fa3-a14f-491cd0d5ce3f","Type":"ContainerStarted","Data":"431c71d22cad4a9818fd34e9546a3cc4f1f1af00bb0911a8509d1727ee3d50a6"} Jul 07 15:41:38.965304 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:38.965262 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:41:39.658017 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:39.657994 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-5c847dcfc4-m4jd9"] Jul 07 15:41:39.659697 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:39.659682 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5c847dcfc4-m4jd9" Jul 07 15:41:39.662522 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:39.662487 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Jul 07 15:41:39.662652 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:39.662600 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Jul 07 15:41:39.663957 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:39.663934 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 07 15:41:39.663957 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:39.663941 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-t2mpm\"" Jul 07 15:41:39.664067 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:39.663943 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 07 15:41:39.668075 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:39.668058 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-6b6cdf54ff-tjwbr"] Jul 07 15:41:39.669806 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:39.669788 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Jul 07 15:41:39.673063 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:39.673042 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5c847dcfc4-m4jd9"] Jul 07 15:41:39.673159 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:39.673141 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-6b6cdf54ff-tjwbr" Jul 07 15:41:39.677211 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:39.676880 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Jul 07 15:41:39.677211 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:39.676970 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 07 15:41:39.677211 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:39.676984 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 07 15:41:39.677442 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:39.677225 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Jul 07 15:41:39.677442 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:39.677329 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Jul 07 15:41:39.677442 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:39.677395 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Jul 07 15:41:39.677663 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:39.677541 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-2cmdl\"" Jul 07 15:41:39.682795 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:39.682765 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-6b6cdf54ff-tjwbr"] Jul 07 15:41:39.687814 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:39.687790 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:41:39.743998 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:39.743966 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/17238627-b0d8-49ae-827e-cbf250ab8146-tmp\") pod \"insights-operator-5c847dcfc4-m4jd9\" (UID: \"17238627-b0d8-49ae-827e-cbf250ab8146\") " pod="openshift-insights/insights-operator-5c847dcfc4-m4jd9" Jul 07 15:41:39.743998 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:39.744001 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/95b55bd6-e205-4969-ac44-d3f67f63f746-metrics-certs\") pod \"router-default-6b6cdf54ff-tjwbr\" (UID: \"95b55bd6-e205-4969-ac44-d3f67f63f746\") " pod="openshift-ingress/router-default-6b6cdf54ff-tjwbr" Jul 07 15:41:39.744244 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:39.744023 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/17238627-b0d8-49ae-827e-cbf250ab8146-service-ca-bundle\") pod \"insights-operator-5c847dcfc4-m4jd9\" (UID: \"17238627-b0d8-49ae-827e-cbf250ab8146\") " pod="openshift-insights/insights-operator-5c847dcfc4-m4jd9" Jul 07 15:41:39.744244 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:39.744044 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/17238627-b0d8-49ae-827e-cbf250ab8146-serving-cert\") pod \"insights-operator-5c847dcfc4-m4jd9\" (UID: \"17238627-b0d8-49ae-827e-cbf250ab8146\") " pod="openshift-insights/insights-operator-5c847dcfc4-m4jd9" Jul 07 15:41:39.744244 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:39.744067 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/17238627-b0d8-49ae-827e-cbf250ab8146-trusted-ca-bundle\") pod \"insights-operator-5c847dcfc4-m4jd9\" (UID: \"17238627-b0d8-49ae-827e-cbf250ab8146\") " pod="openshift-insights/insights-operator-5c847dcfc4-m4jd9" Jul 07 15:41:39.744405 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:39.744233 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/17238627-b0d8-49ae-827e-cbf250ab8146-snapshots\") pod \"insights-operator-5c847dcfc4-m4jd9\" (UID: \"17238627-b0d8-49ae-827e-cbf250ab8146\") " pod="openshift-insights/insights-operator-5c847dcfc4-m4jd9" Jul 07 15:41:39.744405 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:39.744332 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/95b55bd6-e205-4969-ac44-d3f67f63f746-default-certificate\") pod \"router-default-6b6cdf54ff-tjwbr\" (UID: \"95b55bd6-e205-4969-ac44-d3f67f63f746\") " pod="openshift-ingress/router-default-6b6cdf54ff-tjwbr" Jul 07 15:41:39.744405 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:39.744365 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/95b55bd6-e205-4969-ac44-d3f67f63f746-service-ca-bundle\") pod \"router-default-6b6cdf54ff-tjwbr\" (UID: \"95b55bd6-e205-4969-ac44-d3f67f63f746\") " pod="openshift-ingress/router-default-6b6cdf54ff-tjwbr" Jul 07 15:41:39.744405 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:39.744391 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-482pp\" (UniqueName: \"kubernetes.io/projected/95b55bd6-e205-4969-ac44-d3f67f63f746-kube-api-access-482pp\") pod \"router-default-6b6cdf54ff-tjwbr\" (UID: \"95b55bd6-e205-4969-ac44-d3f67f63f746\") " pod="openshift-ingress/router-default-6b6cdf54ff-tjwbr" Jul 07 15:41:39.744584 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:39.744419 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/95b55bd6-e205-4969-ac44-d3f67f63f746-stats-auth\") pod \"router-default-6b6cdf54ff-tjwbr\" (UID: \"95b55bd6-e205-4969-ac44-d3f67f63f746\") " pod="openshift-ingress/router-default-6b6cdf54ff-tjwbr" Jul 07 15:41:39.744636 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:39.744575 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rsk5d\" (UniqueName: \"kubernetes.io/projected/17238627-b0d8-49ae-827e-cbf250ab8146-kube-api-access-rsk5d\") pod \"insights-operator-5c847dcfc4-m4jd9\" (UID: \"17238627-b0d8-49ae-827e-cbf250ab8146\") " pod="openshift-insights/insights-operator-5c847dcfc4-m4jd9" Jul 07 15:41:39.775372 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:39.775334 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:41:39.845641 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:39.845610 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/17238627-b0d8-49ae-827e-cbf250ab8146-tmp\") pod \"insights-operator-5c847dcfc4-m4jd9\" (UID: \"17238627-b0d8-49ae-827e-cbf250ab8146\") " pod="openshift-insights/insights-operator-5c847dcfc4-m4jd9" Jul 07 15:41:39.845802 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:39.845653 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/95b55bd6-e205-4969-ac44-d3f67f63f746-metrics-certs\") pod \"router-default-6b6cdf54ff-tjwbr\" (UID: \"95b55bd6-e205-4969-ac44-d3f67f63f746\") " pod="openshift-ingress/router-default-6b6cdf54ff-tjwbr" Jul 07 15:41:39.845802 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:39.845686 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/17238627-b0d8-49ae-827e-cbf250ab8146-service-ca-bundle\") pod \"insights-operator-5c847dcfc4-m4jd9\" (UID: \"17238627-b0d8-49ae-827e-cbf250ab8146\") " pod="openshift-insights/insights-operator-5c847dcfc4-m4jd9" Jul 07 15:41:39.845802 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:39.845708 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/17238627-b0d8-49ae-827e-cbf250ab8146-serving-cert\") pod \"insights-operator-5c847dcfc4-m4jd9\" (UID: \"17238627-b0d8-49ae-827e-cbf250ab8146\") " pod="openshift-insights/insights-operator-5c847dcfc4-m4jd9" Jul 07 15:41:39.845802 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:39.845735 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/17238627-b0d8-49ae-827e-cbf250ab8146-trusted-ca-bundle\") pod \"insights-operator-5c847dcfc4-m4jd9\" (UID: \"17238627-b0d8-49ae-827e-cbf250ab8146\") " pod="openshift-insights/insights-operator-5c847dcfc4-m4jd9" Jul 07 15:41:39.845802 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:39.845793 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/17238627-b0d8-49ae-827e-cbf250ab8146-snapshots\") pod \"insights-operator-5c847dcfc4-m4jd9\" (UID: \"17238627-b0d8-49ae-827e-cbf250ab8146\") " pod="openshift-insights/insights-operator-5c847dcfc4-m4jd9" Jul 07 15:41:39.846122 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:39.845814 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/95b55bd6-e205-4969-ac44-d3f67f63f746-default-certificate\") pod \"router-default-6b6cdf54ff-tjwbr\" (UID: \"95b55bd6-e205-4969-ac44-d3f67f63f746\") " pod="openshift-ingress/router-default-6b6cdf54ff-tjwbr" Jul 07 15:41:39.846122 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:39.845840 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/95b55bd6-e205-4969-ac44-d3f67f63f746-service-ca-bundle\") pod \"router-default-6b6cdf54ff-tjwbr\" (UID: \"95b55bd6-e205-4969-ac44-d3f67f63f746\") " pod="openshift-ingress/router-default-6b6cdf54ff-tjwbr" Jul 07 15:41:39.846122 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:39.845866 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-482pp\" (UniqueName: \"kubernetes.io/projected/95b55bd6-e205-4969-ac44-d3f67f63f746-kube-api-access-482pp\") pod \"router-default-6b6cdf54ff-tjwbr\" (UID: \"95b55bd6-e205-4969-ac44-d3f67f63f746\") " pod="openshift-ingress/router-default-6b6cdf54ff-tjwbr" Jul 07 15:41:39.846122 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:39.845889 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/95b55bd6-e205-4969-ac44-d3f67f63f746-stats-auth\") pod \"router-default-6b6cdf54ff-tjwbr\" (UID: \"95b55bd6-e205-4969-ac44-d3f67f63f746\") " pod="openshift-ingress/router-default-6b6cdf54ff-tjwbr" Jul 07 15:41:39.846122 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:39.845950 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rsk5d\" (UniqueName: \"kubernetes.io/projected/17238627-b0d8-49ae-827e-cbf250ab8146-kube-api-access-rsk5d\") pod \"insights-operator-5c847dcfc4-m4jd9\" (UID: \"17238627-b0d8-49ae-827e-cbf250ab8146\") " pod="openshift-insights/insights-operator-5c847dcfc4-m4jd9" Jul 07 15:41:39.846122 ip-10-0-132-65 kubenswrapper[2565]: E0707 15:41:39.846028 2565 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 07 15:41:39.846122 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:39.846054 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/17238627-b0d8-49ae-827e-cbf250ab8146-tmp\") pod \"insights-operator-5c847dcfc4-m4jd9\" (UID: \"17238627-b0d8-49ae-827e-cbf250ab8146\") " pod="openshift-insights/insights-operator-5c847dcfc4-m4jd9" Jul 07 15:41:39.846122 ip-10-0-132-65 kubenswrapper[2565]: E0707 15:41:39.846090 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/95b55bd6-e205-4969-ac44-d3f67f63f746-metrics-certs podName:95b55bd6-e205-4969-ac44-d3f67f63f746 nodeName:}" failed. No retries permitted until 2026-07-07 15:41:40.346070672 +0000 UTC m=+39.302648125 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/95b55bd6-e205-4969-ac44-d3f67f63f746-metrics-certs") pod "router-default-6b6cdf54ff-tjwbr" (UID: "95b55bd6-e205-4969-ac44-d3f67f63f746") : secret "router-metrics-certs-default" not found Jul 07 15:41:39.846542 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:39.846381 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/17238627-b0d8-49ae-827e-cbf250ab8146-service-ca-bundle\") pod \"insights-operator-5c847dcfc4-m4jd9\" (UID: \"17238627-b0d8-49ae-827e-cbf250ab8146\") " pod="openshift-insights/insights-operator-5c847dcfc4-m4jd9" Jul 07 15:41:39.846542 ip-10-0-132-65 kubenswrapper[2565]: E0707 15:41:39.846518 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/95b55bd6-e205-4969-ac44-d3f67f63f746-service-ca-bundle podName:95b55bd6-e205-4969-ac44-d3f67f63f746 nodeName:}" failed. No retries permitted until 2026-07-07 15:41:40.346483766 +0000 UTC m=+39.303061212 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/95b55bd6-e205-4969-ac44-d3f67f63f746-service-ca-bundle") pod "router-default-6b6cdf54ff-tjwbr" (UID: "95b55bd6-e205-4969-ac44-d3f67f63f746") : configmap references non-existent config key: service-ca.crt Jul 07 15:41:39.846976 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:39.846946 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/17238627-b0d8-49ae-827e-cbf250ab8146-trusted-ca-bundle\") pod \"insights-operator-5c847dcfc4-m4jd9\" (UID: \"17238627-b0d8-49ae-827e-cbf250ab8146\") " pod="openshift-insights/insights-operator-5c847dcfc4-m4jd9" Jul 07 15:41:39.847087 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:39.847028 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/17238627-b0d8-49ae-827e-cbf250ab8146-snapshots\") pod \"insights-operator-5c847dcfc4-m4jd9\" (UID: \"17238627-b0d8-49ae-827e-cbf250ab8146\") " pod="openshift-insights/insights-operator-5c847dcfc4-m4jd9" Jul 07 15:41:39.850031 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:39.850006 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/17238627-b0d8-49ae-827e-cbf250ab8146-serving-cert\") pod \"insights-operator-5c847dcfc4-m4jd9\" (UID: \"17238627-b0d8-49ae-827e-cbf250ab8146\") " pod="openshift-insights/insights-operator-5c847dcfc4-m4jd9" Jul 07 15:41:39.850160 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:39.850141 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/95b55bd6-e205-4969-ac44-d3f67f63f746-default-certificate\") pod \"router-default-6b6cdf54ff-tjwbr\" (UID: \"95b55bd6-e205-4969-ac44-d3f67f63f746\") " pod="openshift-ingress/router-default-6b6cdf54ff-tjwbr" Jul 07 15:41:39.850365 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:39.850348 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/95b55bd6-e205-4969-ac44-d3f67f63f746-stats-auth\") pod \"router-default-6b6cdf54ff-tjwbr\" (UID: \"95b55bd6-e205-4969-ac44-d3f67f63f746\") " pod="openshift-ingress/router-default-6b6cdf54ff-tjwbr" Jul 07 15:41:39.861412 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:39.861389 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-482pp\" (UniqueName: \"kubernetes.io/projected/95b55bd6-e205-4969-ac44-d3f67f63f746-kube-api-access-482pp\") pod \"router-default-6b6cdf54ff-tjwbr\" (UID: \"95b55bd6-e205-4969-ac44-d3f67f63f746\") " pod="openshift-ingress/router-default-6b6cdf54ff-tjwbr" Jul 07 15:41:39.861741 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:39.861718 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rsk5d\" (UniqueName: \"kubernetes.io/projected/17238627-b0d8-49ae-827e-cbf250ab8146-kube-api-access-rsk5d\") pod \"insights-operator-5c847dcfc4-m4jd9\" (UID: \"17238627-b0d8-49ae-827e-cbf250ab8146\") " pod="openshift-insights/insights-operator-5c847dcfc4-m4jd9" Jul 07 15:41:39.944404 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:39.944328 2565 generic.go:356] "Generic (PLEG): container finished" podID="d3ca04a5-3622-461f-9b02-0220e1c17814" containerID="fa417e1922d30691f8a4ad13f2cac4a1a39a9165014a59f6fd7a7fc39972012b" exitCode=0 Jul 07 15:41:39.944404 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:39.944391 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-f7x65" event={"ID":"d3ca04a5-3622-461f-9b02-0220e1c17814","Type":"ContainerDied","Data":"fa417e1922d30691f8a4ad13f2cac4a1a39a9165014a59f6fd7a7fc39972012b"} Jul 07 15:41:39.968841 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:39.968804 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5c847dcfc4-m4jd9" Jul 07 15:41:40.025124 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:40.025081 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:41:40.126874 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:40.126832 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:41:40.227172 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:40.227080 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:41:40.349990 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:40.349952 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/95b55bd6-e205-4969-ac44-d3f67f63f746-service-ca-bundle\") pod \"router-default-6b6cdf54ff-tjwbr\" (UID: \"95b55bd6-e205-4969-ac44-d3f67f63f746\") " pod="openshift-ingress/router-default-6b6cdf54ff-tjwbr" Jul 07 15:41:40.350178 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:40.350066 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/95b55bd6-e205-4969-ac44-d3f67f63f746-metrics-certs\") pod \"router-default-6b6cdf54ff-tjwbr\" (UID: \"95b55bd6-e205-4969-ac44-d3f67f63f746\") " pod="openshift-ingress/router-default-6b6cdf54ff-tjwbr" Jul 07 15:41:40.350178 ip-10-0-132-65 kubenswrapper[2565]: E0707 15:41:40.350119 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/95b55bd6-e205-4969-ac44-d3f67f63f746-service-ca-bundle podName:95b55bd6-e205-4969-ac44-d3f67f63f746 nodeName:}" failed. No retries permitted until 2026-07-07 15:41:41.350095225 +0000 UTC m=+40.306672673 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/95b55bd6-e205-4969-ac44-d3f67f63f746-service-ca-bundle") pod "router-default-6b6cdf54ff-tjwbr" (UID: "95b55bd6-e205-4969-ac44-d3f67f63f746") : configmap references non-existent config key: service-ca.crt Jul 07 15:41:40.350295 ip-10-0-132-65 kubenswrapper[2565]: E0707 15:41:40.350185 2565 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 07 15:41:40.350295 ip-10-0-132-65 kubenswrapper[2565]: E0707 15:41:40.350238 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/95b55bd6-e205-4969-ac44-d3f67f63f746-metrics-certs podName:95b55bd6-e205-4969-ac44-d3f67f63f746 nodeName:}" failed. No retries permitted until 2026-07-07 15:41:41.350222141 +0000 UTC m=+40.306799588 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/95b55bd6-e205-4969-ac44-d3f67f63f746-metrics-certs") pod "router-default-6b6cdf54ff-tjwbr" (UID: "95b55bd6-e205-4969-ac44-d3f67f63f746") : secret "router-metrics-certs-default" not found Jul 07 15:41:40.641909 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:40.641879 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:41:40.650731 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:40.650711 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5c847dcfc4-m4jd9"] Jul 07 15:41:40.652700 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:40.652681 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cb4ddf17-3bbe-4d75-9510-99f4a61da167-metrics-tls\") pod \"dns-default-qhqsf\" (UID: \"cb4ddf17-3bbe-4d75-9510-99f4a61da167\") " pod="openshift-dns/dns-default-qhqsf" Jul 07 15:41:40.652840 ip-10-0-132-65 kubenswrapper[2565]: E0707 15:41:40.652826 2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 07 15:41:40.652890 ip-10-0-132-65 kubenswrapper[2565]: E0707 15:41:40.652883 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb4ddf17-3bbe-4d75-9510-99f4a61da167-metrics-tls podName:cb4ddf17-3bbe-4d75-9510-99f4a61da167 nodeName:}" failed. No retries permitted until 2026-07-07 15:41:44.65286656 +0000 UTC m=+43.609444007 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/cb4ddf17-3bbe-4d75-9510-99f4a61da167-metrics-tls") pod "dns-default-qhqsf" (UID: "cb4ddf17-3bbe-4d75-9510-99f4a61da167") : secret "dns-default-metrics-tls" not found Jul 07 15:41:40.656088 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:40.656061 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17238627_b0d8_49ae_827e_cbf250ab8146.slice/crio-00061f0d479b8176e81d75be37f57debce2de5574121d1fb11cb1baf122ea141 WatchSource:0}: Error finding container 00061f0d479b8176e81d75be37f57debce2de5574121d1fb11cb1baf122ea141: Status 404 returned error can't find the container with id 00061f0d479b8176e81d75be37f57debce2de5574121d1fb11cb1baf122ea141 Jul 07 15:41:40.913891 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:40.913863 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:41:40.950250 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:40.950213 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-f7x65" event={"ID":"d3ca04a5-3622-461f-9b02-0220e1c17814","Type":"ContainerStarted","Data":"7e216723b61e0e09782c0049cfd64bc6b2ebb7eaf81c9b590a197ed3614afd5c"} Jul 07 15:41:40.951442 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:40.951419 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5c847dcfc4-m4jd9" event={"ID":"17238627-b0d8-49ae-827e-cbf250ab8146","Type":"ContainerStarted","Data":"00061f0d479b8176e81d75be37f57debce2de5574121d1fb11cb1baf122ea141"} Jul 07 15:41:40.991704 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:40.991652 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-f7x65" podStartSLOduration=9.166501066 podStartE2EDuration="39.991633841s" totalImagesPullingTime="30.825132775s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 15:41:01 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:41:40.991023896 +0000 UTC m=+39.947601386" watchObservedRunningTime="2026-07-07 15:41:40.991633841 +0000 UTC m=+39.948211308" Jul 07 15:41:41.235362 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:41.235261 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:41:41.346656 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:41.346624 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:41:41.357645 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:41.357620 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/95b55bd6-e205-4969-ac44-d3f67f63f746-metrics-certs\") pod \"router-default-6b6cdf54ff-tjwbr\" (UID: \"95b55bd6-e205-4969-ac44-d3f67f63f746\") " pod="openshift-ingress/router-default-6b6cdf54ff-tjwbr" Jul 07 15:41:41.357720 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:41.357673 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/95b55bd6-e205-4969-ac44-d3f67f63f746-service-ca-bundle\") pod \"router-default-6b6cdf54ff-tjwbr\" (UID: \"95b55bd6-e205-4969-ac44-d3f67f63f746\") " pod="openshift-ingress/router-default-6b6cdf54ff-tjwbr" Jul 07 15:41:41.357788 ip-10-0-132-65 kubenswrapper[2565]: E0707 15:41:41.357776 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/95b55bd6-e205-4969-ac44-d3f67f63f746-service-ca-bundle podName:95b55bd6-e205-4969-ac44-d3f67f63f746 nodeName:}" failed. No retries permitted until 2026-07-07 15:41:43.357761764 +0000 UTC m=+42.314339210 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/95b55bd6-e205-4969-ac44-d3f67f63f746-service-ca-bundle") pod "router-default-6b6cdf54ff-tjwbr" (UID: "95b55bd6-e205-4969-ac44-d3f67f63f746") : configmap references non-existent config key: service-ca.crt Jul 07 15:41:41.357852 ip-10-0-132-65 kubenswrapper[2565]: E0707 15:41:41.357780 2565 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 07 15:41:41.357852 ip-10-0-132-65 kubenswrapper[2565]: E0707 15:41:41.357841 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/95b55bd6-e205-4969-ac44-d3f67f63f746-metrics-certs podName:95b55bd6-e205-4969-ac44-d3f67f63f746 nodeName:}" failed. No retries permitted until 2026-07-07 15:41:43.357824257 +0000 UTC m=+42.314401705 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/95b55bd6-e205-4969-ac44-d3f67f63f746-metrics-certs") pod "router-default-6b6cdf54ff-tjwbr" (UID: "95b55bd6-e205-4969-ac44-d3f67f63f746") : secret "router-metrics-certs-default" not found Jul 07 15:41:41.955421 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:41.955387 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-t6xpc" event={"ID":"4097f76e-a65f-4fa3-a14f-491cd0d5ce3f","Type":"ContainerStarted","Data":"d0715dba6f3b2171e825abcc9ff420e9e0940f7f95ef5e749cd2f240d1f93700"} Jul 07 15:41:41.978463 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:41.978399 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-t6xpc" podStartSLOduration=3.409654783 podStartE2EDuration="5.978376948s" totalImagesPullingTime="2.568722165s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:41:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:41:41.976667813 +0000 UTC m=+40.933245293" watchObservedRunningTime="2026-07-07 15:41:41.978376948 +0000 UTC m=+40.934954417" Jul 07 15:41:43.372757 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:43.372709 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/95b55bd6-e205-4969-ac44-d3f67f63f746-service-ca-bundle\") pod \"router-default-6b6cdf54ff-tjwbr\" (UID: \"95b55bd6-e205-4969-ac44-d3f67f63f746\") " pod="openshift-ingress/router-default-6b6cdf54ff-tjwbr" Jul 07 15:41:43.373186 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:43.372816 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/95b55bd6-e205-4969-ac44-d3f67f63f746-metrics-certs\") pod \"router-default-6b6cdf54ff-tjwbr\" (UID: \"95b55bd6-e205-4969-ac44-d3f67f63f746\") " pod="openshift-ingress/router-default-6b6cdf54ff-tjwbr" Jul 07 15:41:43.373186 ip-10-0-132-65 kubenswrapper[2565]: E0707 15:41:43.372923 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/95b55bd6-e205-4969-ac44-d3f67f63f746-service-ca-bundle podName:95b55bd6-e205-4969-ac44-d3f67f63f746 nodeName:}" failed. No retries permitted until 2026-07-07 15:41:47.372904196 +0000 UTC m=+46.329481641 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/95b55bd6-e205-4969-ac44-d3f67f63f746-service-ca-bundle") pod "router-default-6b6cdf54ff-tjwbr" (UID: "95b55bd6-e205-4969-ac44-d3f67f63f746") : configmap references non-existent config key: service-ca.crt Jul 07 15:41:43.373186 ip-10-0-132-65 kubenswrapper[2565]: E0707 15:41:43.372937 2565 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 07 15:41:43.373186 ip-10-0-132-65 kubenswrapper[2565]: E0707 15:41:43.373005 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/95b55bd6-e205-4969-ac44-d3f67f63f746-metrics-certs podName:95b55bd6-e205-4969-ac44-d3f67f63f746 nodeName:}" failed. No retries permitted until 2026-07-07 15:41:47.372988578 +0000 UTC m=+46.329566025 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/95b55bd6-e205-4969-ac44-d3f67f63f746-metrics-certs") pod "router-default-6b6cdf54ff-tjwbr" (UID: "95b55bd6-e205-4969-ac44-d3f67f63f746") : secret "router-metrics-certs-default" not found Jul 07 15:41:43.961990 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:43.961953 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5c847dcfc4-m4jd9" event={"ID":"17238627-b0d8-49ae-827e-cbf250ab8146","Type":"ContainerStarted","Data":"493b9985c3dccd03b09ee1f55cdcc36a9e73553ac48231ed8c57e2eeb272f9cc"} Jul 07 15:41:43.983696 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:43.983638 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-5c847dcfc4-m4jd9" podStartSLOduration=1.894435164 podStartE2EDuration="4.983622829s" totalImagesPullingTime="3.089187665s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:41:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:41:43.982890432 +0000 UTC m=+42.939467899" watchObservedRunningTime="2026-07-07 15:41:43.983622829 +0000 UTC m=+42.940200296" Jul 07 15:41:44.682524 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:44.682475 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cb4ddf17-3bbe-4d75-9510-99f4a61da167-metrics-tls\") pod \"dns-default-qhqsf\" (UID: \"cb4ddf17-3bbe-4d75-9510-99f4a61da167\") " pod="openshift-dns/dns-default-qhqsf" Jul 07 15:41:44.682932 ip-10-0-132-65 kubenswrapper[2565]: E0707 15:41:44.682660 2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 07 15:41:44.682932 ip-10-0-132-65 kubenswrapper[2565]: E0707 15:41:44.682740 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb4ddf17-3bbe-4d75-9510-99f4a61da167-metrics-tls podName:cb4ddf17-3bbe-4d75-9510-99f4a61da167 nodeName:}" failed. No retries permitted until 2026-07-07 15:41:52.682718339 +0000 UTC m=+51.639295787 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/cb4ddf17-3bbe-4d75-9510-99f4a61da167-metrics-tls") pod "dns-default-qhqsf" (UID: "cb4ddf17-3bbe-4d75-9510-99f4a61da167") : secret "dns-default-metrics-tls" not found Jul 07 15:41:44.822049 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:44.822021 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-vthmh"] Jul 07 15:41:44.844963 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:44.844936 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-vthmh"] Jul 07 15:41:44.845108 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:44.845056 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-vthmh" Jul 07 15:41:44.847873 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:44.847850 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 07 15:41:44.848120 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:44.848102 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 07 15:41:44.849310 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:44.849293 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-4xbb9\"" Jul 07 15:41:44.859089 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:44.859068 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:41:44.949703 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:44.949621 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:41:44.985192 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:44.985160 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/12101a9b-a592-4514-9eaf-513f0feeb2e9-crio-socket\") pod \"insights-runtime-extractor-vthmh\" (UID: \"12101a9b-a592-4514-9eaf-513f0feeb2e9\") " pod="openshift-insights/insights-runtime-extractor-vthmh" Jul 07 15:41:44.985192 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:44.985190 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/12101a9b-a592-4514-9eaf-513f0feeb2e9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vthmh\" (UID: \"12101a9b-a592-4514-9eaf-513f0feeb2e9\") " pod="openshift-insights/insights-runtime-extractor-vthmh" Jul 07 15:41:44.985381 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:44.985213 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/12101a9b-a592-4514-9eaf-513f0feeb2e9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vthmh\" (UID: \"12101a9b-a592-4514-9eaf-513f0feeb2e9\") " pod="openshift-insights/insights-runtime-extractor-vthmh" Jul 07 15:41:44.985381 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:44.985307 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nf4kn\" (UniqueName: \"kubernetes.io/projected/12101a9b-a592-4514-9eaf-513f0feeb2e9-kube-api-access-nf4kn\") pod \"insights-runtime-extractor-vthmh\" (UID: \"12101a9b-a592-4514-9eaf-513f0feeb2e9\") " pod="openshift-insights/insights-runtime-extractor-vthmh" Jul 07 15:41:44.985381 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:44.985350 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/12101a9b-a592-4514-9eaf-513f0feeb2e9-data-volume\") pod \"insights-runtime-extractor-vthmh\" (UID: \"12101a9b-a592-4514-9eaf-513f0feeb2e9\") " pod="openshift-insights/insights-runtime-extractor-vthmh" Jul 07 15:41:45.086122 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:45.086070 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/12101a9b-a592-4514-9eaf-513f0feeb2e9-crio-socket\") pod \"insights-runtime-extractor-vthmh\" (UID: \"12101a9b-a592-4514-9eaf-513f0feeb2e9\") " pod="openshift-insights/insights-runtime-extractor-vthmh" Jul 07 15:41:45.086297 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:45.086132 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/12101a9b-a592-4514-9eaf-513f0feeb2e9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vthmh\" (UID: \"12101a9b-a592-4514-9eaf-513f0feeb2e9\") " pod="openshift-insights/insights-runtime-extractor-vthmh" Jul 07 15:41:45.086297 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:45.086161 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/12101a9b-a592-4514-9eaf-513f0feeb2e9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vthmh\" (UID: \"12101a9b-a592-4514-9eaf-513f0feeb2e9\") " pod="openshift-insights/insights-runtime-extractor-vthmh" Jul 07 15:41:45.086297 ip-10-0-132-65 kubenswrapper[2565]: E0707 15:41:45.086226 2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 07 15:41:45.086297 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:45.086262 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nf4kn\" (UniqueName: \"kubernetes.io/projected/12101a9b-a592-4514-9eaf-513f0feeb2e9-kube-api-access-nf4kn\") pod \"insights-runtime-extractor-vthmh\" (UID: \"12101a9b-a592-4514-9eaf-513f0feeb2e9\") " pod="openshift-insights/insights-runtime-extractor-vthmh" Jul 07 15:41:45.086297 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:45.086270 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/12101a9b-a592-4514-9eaf-513f0feeb2e9-crio-socket\") pod \"insights-runtime-extractor-vthmh\" (UID: \"12101a9b-a592-4514-9eaf-513f0feeb2e9\") " pod="openshift-insights/insights-runtime-extractor-vthmh" Jul 07 15:41:45.086297 ip-10-0-132-65 kubenswrapper[2565]: E0707 15:41:45.086277 2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 07 15:41:45.086551 ip-10-0-132-65 kubenswrapper[2565]: E0707 15:41:45.086301 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/12101a9b-a592-4514-9eaf-513f0feeb2e9-kube-rbac-proxy-cm podName:12101a9b-a592-4514-9eaf-513f0feeb2e9 nodeName:}" failed. No retries permitted until 2026-07-07 15:41:45.586280284 +0000 UTC m=+44.542857730 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/12101a9b-a592-4514-9eaf-513f0feeb2e9-kube-rbac-proxy-cm") pod "insights-runtime-extractor-vthmh" (UID: "12101a9b-a592-4514-9eaf-513f0feeb2e9") : configmap "kube-rbac-proxy" not found Jul 07 15:41:45.086551 ip-10-0-132-65 kubenswrapper[2565]: E0707 15:41:45.086341 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/12101a9b-a592-4514-9eaf-513f0feeb2e9-insights-runtime-extractor-tls podName:12101a9b-a592-4514-9eaf-513f0feeb2e9 nodeName:}" failed. No retries permitted until 2026-07-07 15:41:45.586328254 +0000 UTC m=+44.542905703 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/12101a9b-a592-4514-9eaf-513f0feeb2e9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vthmh" (UID: "12101a9b-a592-4514-9eaf-513f0feeb2e9") : secret "insights-runtime-extractor-tls" not found Jul 07 15:41:45.086551 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:45.086387 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/12101a9b-a592-4514-9eaf-513f0feeb2e9-data-volume\") pod \"insights-runtime-extractor-vthmh\" (UID: \"12101a9b-a592-4514-9eaf-513f0feeb2e9\") " pod="openshift-insights/insights-runtime-extractor-vthmh" Jul 07 15:41:45.086676 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:45.086649 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/12101a9b-a592-4514-9eaf-513f0feeb2e9-data-volume\") pod \"insights-runtime-extractor-vthmh\" (UID: \"12101a9b-a592-4514-9eaf-513f0feeb2e9\") " pod="openshift-insights/insights-runtime-extractor-vthmh" Jul 07 15:41:45.098290 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:45.098262 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nf4kn\" (UniqueName: \"kubernetes.io/projected/12101a9b-a592-4514-9eaf-513f0feeb2e9-kube-api-access-nf4kn\") pod \"insights-runtime-extractor-vthmh\" (UID: \"12101a9b-a592-4514-9eaf-513f0feeb2e9\") " pod="openshift-insights/insights-runtime-extractor-vthmh" Jul 07 15:41:45.590094 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:45.590045 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/12101a9b-a592-4514-9eaf-513f0feeb2e9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vthmh\" (UID: \"12101a9b-a592-4514-9eaf-513f0feeb2e9\") " pod="openshift-insights/insights-runtime-extractor-vthmh" Jul 07 15:41:45.590094 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:45.590092 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/12101a9b-a592-4514-9eaf-513f0feeb2e9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vthmh\" (UID: \"12101a9b-a592-4514-9eaf-513f0feeb2e9\") " pod="openshift-insights/insights-runtime-extractor-vthmh" Jul 07 15:41:45.590345 ip-10-0-132-65 kubenswrapper[2565]: E0707 15:41:45.590169 2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 07 15:41:45.590345 ip-10-0-132-65 kubenswrapper[2565]: E0707 15:41:45.590228 2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 07 15:41:45.590345 ip-10-0-132-65 kubenswrapper[2565]: E0707 15:41:45.590242 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/12101a9b-a592-4514-9eaf-513f0feeb2e9-kube-rbac-proxy-cm podName:12101a9b-a592-4514-9eaf-513f0feeb2e9 nodeName:}" failed. No retries permitted until 2026-07-07 15:41:46.590226019 +0000 UTC m=+45.546803468 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/12101a9b-a592-4514-9eaf-513f0feeb2e9-kube-rbac-proxy-cm") pod "insights-runtime-extractor-vthmh" (UID: "12101a9b-a592-4514-9eaf-513f0feeb2e9") : configmap "kube-rbac-proxy" not found Jul 07 15:41:45.590345 ip-10-0-132-65 kubenswrapper[2565]: E0707 15:41:45.590277 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/12101a9b-a592-4514-9eaf-513f0feeb2e9-insights-runtime-extractor-tls podName:12101a9b-a592-4514-9eaf-513f0feeb2e9 nodeName:}" failed. No retries permitted until 2026-07-07 15:41:46.590262697 +0000 UTC m=+45.546840142 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/12101a9b-a592-4514-9eaf-513f0feeb2e9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vthmh" (UID: "12101a9b-a592-4514-9eaf-513f0feeb2e9") : secret "insights-runtime-extractor-tls" not found Jul 07 15:41:46.597531 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:46.597476 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/12101a9b-a592-4514-9eaf-513f0feeb2e9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vthmh\" (UID: \"12101a9b-a592-4514-9eaf-513f0feeb2e9\") " pod="openshift-insights/insights-runtime-extractor-vthmh" Jul 07 15:41:46.597531 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:46.597536 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/12101a9b-a592-4514-9eaf-513f0feeb2e9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vthmh\" (UID: \"12101a9b-a592-4514-9eaf-513f0feeb2e9\") " pod="openshift-insights/insights-runtime-extractor-vthmh" Jul 07 15:41:46.598027 ip-10-0-132-65 kubenswrapper[2565]: E0707 15:41:46.597665 2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 07 15:41:46.598027 ip-10-0-132-65 kubenswrapper[2565]: E0707 15:41:46.597675 2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 07 15:41:46.598027 ip-10-0-132-65 kubenswrapper[2565]: E0707 15:41:46.597728 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/12101a9b-a592-4514-9eaf-513f0feeb2e9-kube-rbac-proxy-cm podName:12101a9b-a592-4514-9eaf-513f0feeb2e9 nodeName:}" failed. No retries permitted until 2026-07-07 15:41:48.597712632 +0000 UTC m=+47.554290077 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/12101a9b-a592-4514-9eaf-513f0feeb2e9-kube-rbac-proxy-cm") pod "insights-runtime-extractor-vthmh" (UID: "12101a9b-a592-4514-9eaf-513f0feeb2e9") : configmap "kube-rbac-proxy" not found Jul 07 15:41:46.598027 ip-10-0-132-65 kubenswrapper[2565]: E0707 15:41:46.597744 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/12101a9b-a592-4514-9eaf-513f0feeb2e9-insights-runtime-extractor-tls podName:12101a9b-a592-4514-9eaf-513f0feeb2e9 nodeName:}" failed. No retries permitted until 2026-07-07 15:41:48.597738232 +0000 UTC m=+47.554315677 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/12101a9b-a592-4514-9eaf-513f0feeb2e9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vthmh" (UID: "12101a9b-a592-4514-9eaf-513f0feeb2e9") : secret "insights-runtime-extractor-tls" not found Jul 07 15:41:46.601963 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:46.601944 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-qggz6_d6c50a3b-0739-431c-80b0-8f42c4b00405/dns-node-resolver/0.log" Jul 07 15:41:47.002388 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:47.002320 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-9fmf8_e5c3e573-02fc-4e63-b475-65ee8b00f34b/node-ca/0.log" Jul 07 15:41:47.404188 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:47.404150 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/95b55bd6-e205-4969-ac44-d3f67f63f746-service-ca-bundle\") pod \"router-default-6b6cdf54ff-tjwbr\" (UID: \"95b55bd6-e205-4969-ac44-d3f67f63f746\") " pod="openshift-ingress/router-default-6b6cdf54ff-tjwbr" Jul 07 15:41:47.404385 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:47.404283 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/95b55bd6-e205-4969-ac44-d3f67f63f746-metrics-certs\") pod \"router-default-6b6cdf54ff-tjwbr\" (UID: \"95b55bd6-e205-4969-ac44-d3f67f63f746\") " pod="openshift-ingress/router-default-6b6cdf54ff-tjwbr" Jul 07 15:41:47.404385 ip-10-0-132-65 kubenswrapper[2565]: E0707 15:41:47.404356 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/95b55bd6-e205-4969-ac44-d3f67f63f746-service-ca-bundle podName:95b55bd6-e205-4969-ac44-d3f67f63f746 nodeName:}" failed. No retries permitted until 2026-07-07 15:41:55.404334078 +0000 UTC m=+54.360911525 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/95b55bd6-e205-4969-ac44-d3f67f63f746-service-ca-bundle") pod "router-default-6b6cdf54ff-tjwbr" (UID: "95b55bd6-e205-4969-ac44-d3f67f63f746") : configmap references non-existent config key: service-ca.crt Jul 07 15:41:47.404541 ip-10-0-132-65 kubenswrapper[2565]: E0707 15:41:47.404391 2565 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 07 15:41:47.404541 ip-10-0-132-65 kubenswrapper[2565]: E0707 15:41:47.404431 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/95b55bd6-e205-4969-ac44-d3f67f63f746-metrics-certs podName:95b55bd6-e205-4969-ac44-d3f67f63f746 nodeName:}" failed. No retries permitted until 2026-07-07 15:41:55.404420107 +0000 UTC m=+54.360997552 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/95b55bd6-e205-4969-ac44-d3f67f63f746-metrics-certs") pod "router-default-6b6cdf54ff-tjwbr" (UID: "95b55bd6-e205-4969-ac44-d3f67f63f746") : secret "router-metrics-certs-default" not found Jul 07 15:41:48.002061 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:48.002018 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-t6xpc_4097f76e-a65f-4fa3-a14f-491cd0d5ce3f/serve-healthcheck-canary/0.log" Jul 07 15:41:48.613773 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:48.613733 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/12101a9b-a592-4514-9eaf-513f0feeb2e9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vthmh\" (UID: \"12101a9b-a592-4514-9eaf-513f0feeb2e9\") " pod="openshift-insights/insights-runtime-extractor-vthmh" Jul 07 15:41:48.613773 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:48.613774 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/12101a9b-a592-4514-9eaf-513f0feeb2e9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vthmh\" (UID: \"12101a9b-a592-4514-9eaf-513f0feeb2e9\") " pod="openshift-insights/insights-runtime-extractor-vthmh" Jul 07 15:41:48.614215 ip-10-0-132-65 kubenswrapper[2565]: E0707 15:41:48.613873 2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 07 15:41:48.614215 ip-10-0-132-65 kubenswrapper[2565]: E0707 15:41:48.613924 2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 07 15:41:48.614215 ip-10-0-132-65 kubenswrapper[2565]: E0707 15:41:48.613940 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/12101a9b-a592-4514-9eaf-513f0feeb2e9-kube-rbac-proxy-cm podName:12101a9b-a592-4514-9eaf-513f0feeb2e9 nodeName:}" failed. No retries permitted until 2026-07-07 15:41:52.613924722 +0000 UTC m=+51.570502167 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/12101a9b-a592-4514-9eaf-513f0feeb2e9-kube-rbac-proxy-cm") pod "insights-runtime-extractor-vthmh" (UID: "12101a9b-a592-4514-9eaf-513f0feeb2e9") : configmap "kube-rbac-proxy" not found Jul 07 15:41:48.614215 ip-10-0-132-65 kubenswrapper[2565]: E0707 15:41:48.613978 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/12101a9b-a592-4514-9eaf-513f0feeb2e9-insights-runtime-extractor-tls podName:12101a9b-a592-4514-9eaf-513f0feeb2e9 nodeName:}" failed. No retries permitted until 2026-07-07 15:41:52.613963572 +0000 UTC m=+51.570541017 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/12101a9b-a592-4514-9eaf-513f0feeb2e9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vthmh" (UID: "12101a9b-a592-4514-9eaf-513f0feeb2e9") : secret "insights-runtime-extractor-tls" not found Jul 07 15:41:49.761955 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:49.761924 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-p6fgv"] Jul 07 15:41:49.801616 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:49.801588 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-68b757fb67-s9kl8"] Jul 07 15:41:49.801764 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:49.801713 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-p6fgv" Jul 07 15:41:49.804358 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:49.804338 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Jul 07 15:41:49.804508 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:49.804483 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Jul 07 15:41:49.804581 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:49.804563 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-lh2m2\"" Jul 07 15:41:49.804644 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:49.804566 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Jul 07 15:41:49.804644 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:49.804617 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Jul 07 15:41:49.820696 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:49.820676 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-p6fgv"] Jul 07 15:41:49.820781 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:49.820705 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-68b757fb67-s9kl8"] Jul 07 15:41:49.820823 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:49.820775 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-68b757fb67-s9kl8" Jul 07 15:41:49.823633 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:49.823614 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Jul 07 15:41:49.823729 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:49.823626 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Jul 07 15:41:49.823729 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:49.823675 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Jul 07 15:41:49.823846 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:49.823732 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Jul 07 15:41:49.823846 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:49.823792 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-dkqnb\"" Jul 07 15:41:49.829427 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:49.829411 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Jul 07 15:41:49.923725 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:49.923699 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/62da0165-d32c-4bba-9844-a49154d765e5-config\") pod \"console-operator-68b757fb67-s9kl8\" (UID: \"62da0165-d32c-4bba-9844-a49154d765e5\") " pod="openshift-console-operator/console-operator-68b757fb67-s9kl8" Jul 07 15:41:49.923725 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:49.923729 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/62da0165-d32c-4bba-9844-a49154d765e5-trusted-ca\") pod \"console-operator-68b757fb67-s9kl8\" (UID: \"62da0165-d32c-4bba-9844-a49154d765e5\") " pod="openshift-console-operator/console-operator-68b757fb67-s9kl8" Jul 07 15:41:49.923967 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:49.923753 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vnpkr\" (UniqueName: \"kubernetes.io/projected/394eb1a2-5a48-4813-b38b-227294ed15ba-kube-api-access-vnpkr\") pod \"kube-storage-version-migrator-operator-846bf668f7-p6fgv\" (UID: \"394eb1a2-5a48-4813-b38b-227294ed15ba\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-p6fgv" Jul 07 15:41:49.923967 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:49.923857 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dk9ck\" (UniqueName: \"kubernetes.io/projected/62da0165-d32c-4bba-9844-a49154d765e5-kube-api-access-dk9ck\") pod \"console-operator-68b757fb67-s9kl8\" (UID: \"62da0165-d32c-4bba-9844-a49154d765e5\") " pod="openshift-console-operator/console-operator-68b757fb67-s9kl8" Jul 07 15:41:49.923967 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:49.923947 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/62da0165-d32c-4bba-9844-a49154d765e5-serving-cert\") pod \"console-operator-68b757fb67-s9kl8\" (UID: \"62da0165-d32c-4bba-9844-a49154d765e5\") " pod="openshift-console-operator/console-operator-68b757fb67-s9kl8" Jul 07 15:41:49.924116 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:49.923993 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/394eb1a2-5a48-4813-b38b-227294ed15ba-serving-cert\") pod \"kube-storage-version-migrator-operator-846bf668f7-p6fgv\" (UID: \"394eb1a2-5a48-4813-b38b-227294ed15ba\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-p6fgv" Jul 07 15:41:49.924116 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:49.924022 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/394eb1a2-5a48-4813-b38b-227294ed15ba-config\") pod \"kube-storage-version-migrator-operator-846bf668f7-p6fgv\" (UID: \"394eb1a2-5a48-4813-b38b-227294ed15ba\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-p6fgv" Jul 07 15:41:50.025478 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:50.025389 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/62da0165-d32c-4bba-9844-a49154d765e5-config\") pod \"console-operator-68b757fb67-s9kl8\" (UID: \"62da0165-d32c-4bba-9844-a49154d765e5\") " pod="openshift-console-operator/console-operator-68b757fb67-s9kl8" Jul 07 15:41:50.025478 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:50.025428 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/62da0165-d32c-4bba-9844-a49154d765e5-trusted-ca\") pod \"console-operator-68b757fb67-s9kl8\" (UID: \"62da0165-d32c-4bba-9844-a49154d765e5\") " pod="openshift-console-operator/console-operator-68b757fb67-s9kl8" Jul 07 15:41:50.025478 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:50.025449 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vnpkr\" (UniqueName: \"kubernetes.io/projected/394eb1a2-5a48-4813-b38b-227294ed15ba-kube-api-access-vnpkr\") pod \"kube-storage-version-migrator-operator-846bf668f7-p6fgv\" (UID: \"394eb1a2-5a48-4813-b38b-227294ed15ba\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-p6fgv" Jul 07 15:41:50.025478 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:50.025481 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dk9ck\" (UniqueName: \"kubernetes.io/projected/62da0165-d32c-4bba-9844-a49154d765e5-kube-api-access-dk9ck\") pod \"console-operator-68b757fb67-s9kl8\" (UID: \"62da0165-d32c-4bba-9844-a49154d765e5\") " pod="openshift-console-operator/console-operator-68b757fb67-s9kl8" Jul 07 15:41:50.025838 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:50.025571 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/62da0165-d32c-4bba-9844-a49154d765e5-serving-cert\") pod \"console-operator-68b757fb67-s9kl8\" (UID: \"62da0165-d32c-4bba-9844-a49154d765e5\") " pod="openshift-console-operator/console-operator-68b757fb67-s9kl8" Jul 07 15:41:50.025838 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:50.025617 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/394eb1a2-5a48-4813-b38b-227294ed15ba-serving-cert\") pod \"kube-storage-version-migrator-operator-846bf668f7-p6fgv\" (UID: \"394eb1a2-5a48-4813-b38b-227294ed15ba\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-p6fgv" Jul 07 15:41:50.025838 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:50.025641 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/394eb1a2-5a48-4813-b38b-227294ed15ba-config\") pod \"kube-storage-version-migrator-operator-846bf668f7-p6fgv\" (UID: \"394eb1a2-5a48-4813-b38b-227294ed15ba\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-p6fgv" Jul 07 15:41:50.026295 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:50.026267 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/62da0165-d32c-4bba-9844-a49154d765e5-config\") pod \"console-operator-68b757fb67-s9kl8\" (UID: \"62da0165-d32c-4bba-9844-a49154d765e5\") " pod="openshift-console-operator/console-operator-68b757fb67-s9kl8" Jul 07 15:41:50.026295 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:50.026293 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/394eb1a2-5a48-4813-b38b-227294ed15ba-config\") pod \"kube-storage-version-migrator-operator-846bf668f7-p6fgv\" (UID: \"394eb1a2-5a48-4813-b38b-227294ed15ba\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-p6fgv" Jul 07 15:41:50.026429 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:50.026334 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/62da0165-d32c-4bba-9844-a49154d765e5-trusted-ca\") pod \"console-operator-68b757fb67-s9kl8\" (UID: \"62da0165-d32c-4bba-9844-a49154d765e5\") " pod="openshift-console-operator/console-operator-68b757fb67-s9kl8" Jul 07 15:41:50.027925 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:50.027904 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/62da0165-d32c-4bba-9844-a49154d765e5-serving-cert\") pod \"console-operator-68b757fb67-s9kl8\" (UID: \"62da0165-d32c-4bba-9844-a49154d765e5\") " pod="openshift-console-operator/console-operator-68b757fb67-s9kl8" Jul 07 15:41:50.028039 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:50.027908 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/394eb1a2-5a48-4813-b38b-227294ed15ba-serving-cert\") pod \"kube-storage-version-migrator-operator-846bf668f7-p6fgv\" (UID: \"394eb1a2-5a48-4813-b38b-227294ed15ba\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-p6fgv" Jul 07 15:41:50.035400 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:50.035369 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vnpkr\" (UniqueName: \"kubernetes.io/projected/394eb1a2-5a48-4813-b38b-227294ed15ba-kube-api-access-vnpkr\") pod \"kube-storage-version-migrator-operator-846bf668f7-p6fgv\" (UID: \"394eb1a2-5a48-4813-b38b-227294ed15ba\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-p6fgv" Jul 07 15:41:50.035515 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:50.035451 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dk9ck\" (UniqueName: \"kubernetes.io/projected/62da0165-d32c-4bba-9844-a49154d765e5-kube-api-access-dk9ck\") pod \"console-operator-68b757fb67-s9kl8\" (UID: \"62da0165-d32c-4bba-9844-a49154d765e5\") " pod="openshift-console-operator/console-operator-68b757fb67-s9kl8" Jul 07 15:41:50.110370 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:50.110337 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-p6fgv" Jul 07 15:41:50.129632 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:50.129596 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-68b757fb67-s9kl8" Jul 07 15:41:50.267064 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:50.267031 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-p6fgv"] Jul 07 15:41:50.270572 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:50.270548 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod394eb1a2_5a48_4813_b38b_227294ed15ba.slice/crio-c43f448fe16cfa6ed138d45ead17b8efccc2cb45023276c268f6c151982383eb WatchSource:0}: Error finding container c43f448fe16cfa6ed138d45ead17b8efccc2cb45023276c268f6c151982383eb: Status 404 returned error can't find the container with id c43f448fe16cfa6ed138d45ead17b8efccc2cb45023276c268f6c151982383eb Jul 07 15:41:50.293169 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:50.293104 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-68b757fb67-s9kl8"] Jul 07 15:41:50.295851 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:50.295829 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62da0165_d32c_4bba_9844_a49154d765e5.slice/crio-d1606e4a0644f5cd342a9b63c08f73cb36a1cdb8ece881bb65d5d0cc00b9648f WatchSource:0}: Error finding container d1606e4a0644f5cd342a9b63c08f73cb36a1cdb8ece881bb65d5d0cc00b9648f: Status 404 returned error can't find the container with id d1606e4a0644f5cd342a9b63c08f73cb36a1cdb8ece881bb65d5d0cc00b9648f Jul 07 15:41:50.979669 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:50.979629 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-p6fgv" event={"ID":"394eb1a2-5a48-4813-b38b-227294ed15ba","Type":"ContainerStarted","Data":"c43f448fe16cfa6ed138d45ead17b8efccc2cb45023276c268f6c151982383eb"} Jul 07 15:41:50.980949 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:50.980918 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-68b757fb67-s9kl8" event={"ID":"62da0165-d32c-4bba-9844-a49154d765e5","Type":"ContainerStarted","Data":"d1606e4a0644f5cd342a9b63c08f73cb36a1cdb8ece881bb65d5d0cc00b9648f"} Jul 07 15:41:51.260137 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:51.260061 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:41:52.651640 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:52.651586 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/12101a9b-a592-4514-9eaf-513f0feeb2e9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vthmh\" (UID: \"12101a9b-a592-4514-9eaf-513f0feeb2e9\") " pod="openshift-insights/insights-runtime-extractor-vthmh" Jul 07 15:41:52.652102 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:52.651714 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/12101a9b-a592-4514-9eaf-513f0feeb2e9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vthmh\" (UID: \"12101a9b-a592-4514-9eaf-513f0feeb2e9\") " pod="openshift-insights/insights-runtime-extractor-vthmh" Jul 07 15:41:52.652102 ip-10-0-132-65 kubenswrapper[2565]: E0707 15:41:52.651766 2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 07 15:41:52.652102 ip-10-0-132-65 kubenswrapper[2565]: E0707 15:41:52.651853 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/12101a9b-a592-4514-9eaf-513f0feeb2e9-insights-runtime-extractor-tls podName:12101a9b-a592-4514-9eaf-513f0feeb2e9 nodeName:}" failed. No retries permitted until 2026-07-07 15:42:00.651829185 +0000 UTC m=+59.608406632 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/12101a9b-a592-4514-9eaf-513f0feeb2e9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vthmh" (UID: "12101a9b-a592-4514-9eaf-513f0feeb2e9") : secret "insights-runtime-extractor-tls" not found Jul 07 15:41:52.652338 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:52.652317 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/12101a9b-a592-4514-9eaf-513f0feeb2e9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vthmh\" (UID: \"12101a9b-a592-4514-9eaf-513f0feeb2e9\") " pod="openshift-insights/insights-runtime-extractor-vthmh" Jul 07 15:41:52.752706 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:52.752661 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cb4ddf17-3bbe-4d75-9510-99f4a61da167-metrics-tls\") pod \"dns-default-qhqsf\" (UID: \"cb4ddf17-3bbe-4d75-9510-99f4a61da167\") " pod="openshift-dns/dns-default-qhqsf" Jul 07 15:41:52.752875 ip-10-0-132-65 kubenswrapper[2565]: E0707 15:41:52.752830 2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 07 15:41:52.752917 ip-10-0-132-65 kubenswrapper[2565]: E0707 15:41:52.752894 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb4ddf17-3bbe-4d75-9510-99f4a61da167-metrics-tls podName:cb4ddf17-3bbe-4d75-9510-99f4a61da167 nodeName:}" failed. No retries permitted until 2026-07-07 15:42:08.752878526 +0000 UTC m=+67.709455970 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/cb4ddf17-3bbe-4d75-9510-99f4a61da167-metrics-tls") pod "dns-default-qhqsf" (UID: "cb4ddf17-3bbe-4d75-9510-99f4a61da167") : secret "dns-default-metrics-tls" not found Jul 07 15:41:52.813986 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:52.813951 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:41:52.894418 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:52.894390 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:41:53.989178 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:53.989137 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-p6fgv" event={"ID":"394eb1a2-5a48-4813-b38b-227294ed15ba","Type":"ContainerStarted","Data":"d211acffe82cddb4ed78d961983111efbfdb31dc08460e53a2c9916c96ad0c12"} Jul 07 15:41:53.990799 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:53.990778 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-s9kl8_62da0165-d32c-4bba-9844-a49154d765e5/console-operator/0.log" Jul 07 15:41:53.990911 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:53.990821 2565 generic.go:356] "Generic (PLEG): container finished" podID="62da0165-d32c-4bba-9844-a49154d765e5" containerID="65343c384107edb4ae2b3f2a0b89a8b0d6c99682fc8121fc7253d00623eb1995" exitCode=255 Jul 07 15:41:53.990911 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:53.990868 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-68b757fb67-s9kl8" event={"ID":"62da0165-d32c-4bba-9844-a49154d765e5","Type":"ContainerDied","Data":"65343c384107edb4ae2b3f2a0b89a8b0d6c99682fc8121fc7253d00623eb1995"} Jul 07 15:41:53.991096 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:53.991083 2565 scope.go:117] "RemoveContainer" containerID="65343c384107edb4ae2b3f2a0b89a8b0d6c99682fc8121fc7253d00623eb1995" Jul 07 15:41:54.007983 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:54.007934 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-p6fgv" podStartSLOduration=1.990229504 podStartE2EDuration="5.007920689s" totalImagesPullingTime="3.017691185s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:41:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:41:54.007420419 +0000 UTC m=+52.963997889" watchObservedRunningTime="2026-07-07 15:41:54.007920689 +0000 UTC m=+52.964498156" Jul 07 15:41:54.708300 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:54.708263 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:41:54.806292 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:54.806258 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:41:54.873316 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:54.873279 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-d57d58cc9-l8sjd"] Jul 07 15:41:54.894541 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:54.894514 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-d57d58cc9-l8sjd"] Jul 07 15:41:54.894705 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:54.894588 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-l8sjd" Jul 07 15:41:54.898329 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:54.898307 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Jul 07 15:41:54.899548 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:54.899523 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-whwcg\"" Jul 07 15:41:54.899664 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:54.899563 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Jul 07 15:41:54.995055 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:54.994984 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-s9kl8_62da0165-d32c-4bba-9844-a49154d765e5/console-operator/1.log" Jul 07 15:41:54.995517 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:54.995387 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-s9kl8_62da0165-d32c-4bba-9844-a49154d765e5/console-operator/0.log" Jul 07 15:41:54.995517 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:54.995430 2565 generic.go:356] "Generic (PLEG): container finished" podID="62da0165-d32c-4bba-9844-a49154d765e5" containerID="ac7cb63b1b61c1458da32332e3b26404b52afe757964b7dc3c5fd41e55bae8fa" exitCode=255 Jul 07 15:41:54.995614 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:54.995516 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-68b757fb67-s9kl8" event={"ID":"62da0165-d32c-4bba-9844-a49154d765e5","Type":"ContainerDied","Data":"ac7cb63b1b61c1458da32332e3b26404b52afe757964b7dc3c5fd41e55bae8fa"} Jul 07 15:41:54.995614 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:54.995565 2565 scope.go:117] "RemoveContainer" containerID="65343c384107edb4ae2b3f2a0b89a8b0d6c99682fc8121fc7253d00623eb1995" Jul 07 15:41:54.995772 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:54.995745 2565 scope.go:117] "RemoveContainer" containerID="ac7cb63b1b61c1458da32332e3b26404b52afe757964b7dc3c5fd41e55bae8fa" Jul 07 15:41:54.995939 ip-10-0-132-65 kubenswrapper[2565]: E0707 15:41:54.995917 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-68b757fb67-s9kl8_openshift-console-operator(62da0165-d32c-4bba-9844-a49154d765e5)\"" pod="openshift-console-operator/console-operator-68b757fb67-s9kl8" podUID="62da0165-d32c-4bba-9844-a49154d765e5" Jul 07 15:41:55.073694 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:55.073664 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qsjzt\" (UniqueName: \"kubernetes.io/projected/0fe973cf-559f-4351-86fc-308c879b0541-kube-api-access-qsjzt\") pod \"migrator-d57d58cc9-l8sjd\" (UID: \"0fe973cf-559f-4351-86fc-308c879b0541\") " pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-l8sjd" Jul 07 15:41:55.174485 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:55.174448 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qsjzt\" (UniqueName: \"kubernetes.io/projected/0fe973cf-559f-4351-86fc-308c879b0541-kube-api-access-qsjzt\") pod \"migrator-d57d58cc9-l8sjd\" (UID: \"0fe973cf-559f-4351-86fc-308c879b0541\") " pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-l8sjd" Jul 07 15:41:55.188882 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:55.188850 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qsjzt\" (UniqueName: \"kubernetes.io/projected/0fe973cf-559f-4351-86fc-308c879b0541-kube-api-access-qsjzt\") pod \"migrator-d57d58cc9-l8sjd\" (UID: \"0fe973cf-559f-4351-86fc-308c879b0541\") " pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-l8sjd" Jul 07 15:41:55.202840 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:55.202817 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-l8sjd" Jul 07 15:41:55.301720 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:55.301687 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:41:55.360228 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:55.360202 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-d57d58cc9-l8sjd"] Jul 07 15:41:55.362281 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:41:55.362252 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fe973cf_559f_4351_86fc_308c879b0541.slice/crio-f288933711396a328b6d0fc4dc70fe63d82dcd86cbfee3de99d7aa0070754e70 WatchSource:0}: Error finding container f288933711396a328b6d0fc4dc70fe63d82dcd86cbfee3de99d7aa0070754e70: Status 404 returned error can't find the container with id f288933711396a328b6d0fc4dc70fe63d82dcd86cbfee3de99d7aa0070754e70 Jul 07 15:41:55.401763 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:55.401730 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:41:55.476929 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:55.476887 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/95b55bd6-e205-4969-ac44-d3f67f63f746-metrics-certs\") pod \"router-default-6b6cdf54ff-tjwbr\" (UID: \"95b55bd6-e205-4969-ac44-d3f67f63f746\") " pod="openshift-ingress/router-default-6b6cdf54ff-tjwbr" Jul 07 15:41:55.477138 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:55.476941 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/95b55bd6-e205-4969-ac44-d3f67f63f746-service-ca-bundle\") pod \"router-default-6b6cdf54ff-tjwbr\" (UID: \"95b55bd6-e205-4969-ac44-d3f67f63f746\") " pod="openshift-ingress/router-default-6b6cdf54ff-tjwbr" Jul 07 15:41:55.477138 ip-10-0-132-65 kubenswrapper[2565]: E0707 15:41:55.477038 2565 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 07 15:41:55.477138 ip-10-0-132-65 kubenswrapper[2565]: E0707 15:41:55.477073 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/95b55bd6-e205-4969-ac44-d3f67f63f746-service-ca-bundle podName:95b55bd6-e205-4969-ac44-d3f67f63f746 nodeName:}" failed. No retries permitted until 2026-07-07 15:42:11.477056059 +0000 UTC m=+70.433633503 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/95b55bd6-e205-4969-ac44-d3f67f63f746-service-ca-bundle") pod "router-default-6b6cdf54ff-tjwbr" (UID: "95b55bd6-e205-4969-ac44-d3f67f63f746") : configmap references non-existent config key: service-ca.crt Jul 07 15:41:55.477138 ip-10-0-132-65 kubenswrapper[2565]: E0707 15:41:55.477095 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/95b55bd6-e205-4969-ac44-d3f67f63f746-metrics-certs podName:95b55bd6-e205-4969-ac44-d3f67f63f746 nodeName:}" failed. No retries permitted until 2026-07-07 15:42:11.477085795 +0000 UTC m=+70.433663253 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/95b55bd6-e205-4969-ac44-d3f67f63f746-metrics-certs") pod "router-default-6b6cdf54ff-tjwbr" (UID: "95b55bd6-e205-4969-ac44-d3f67f63f746") : secret "router-metrics-certs-default" not found Jul 07 15:41:55.779405 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:55.779373 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:41:55.877234 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:55.877201 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:41:55.998967 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:55.998938 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-s9kl8_62da0165-d32c-4bba-9844-a49154d765e5/console-operator/1.log" Jul 07 15:41:55.999366 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:55.999304 2565 scope.go:117] "RemoveContainer" containerID="ac7cb63b1b61c1458da32332e3b26404b52afe757964b7dc3c5fd41e55bae8fa" Jul 07 15:41:55.999503 ip-10-0-132-65 kubenswrapper[2565]: E0707 15:41:55.999466 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-68b757fb67-s9kl8_openshift-console-operator(62da0165-d32c-4bba-9844-a49154d765e5)\"" pod="openshift-console-operator/console-operator-68b757fb67-s9kl8" podUID="62da0165-d32c-4bba-9844-a49154d765e5" Jul 07 15:41:56.000103 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:56.000083 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-l8sjd" event={"ID":"0fe973cf-559f-4351-86fc-308c879b0541","Type":"ContainerStarted","Data":"f288933711396a328b6d0fc4dc70fe63d82dcd86cbfee3de99d7aa0070754e70"} Jul 07 15:41:57.007956 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:57.007923 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:41:57.094807 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:57.094775 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:41:57.194748 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:57.194716 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:41:57.533051 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:57.533016 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:41:57.646907 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:57.646877 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:41:57.998671 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:57.998638 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:41:58.007483 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:58.007460 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-l8sjd" event={"ID":"0fe973cf-559f-4351-86fc-308c879b0541","Type":"ContainerStarted","Data":"5a1371f80c2b476c870187b86964caf0e2275f73b8b7288cf67a5ea903384574"} Jul 07 15:41:58.007592 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:58.007532 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-l8sjd" event={"ID":"0fe973cf-559f-4351-86fc-308c879b0541","Type":"ContainerStarted","Data":"4fe0c1fbbd5b42cfb0b4b9353827f924766f26dda64a8a31c00ef3f5d6316227"} Jul 07 15:41:58.026698 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:58.026649 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-l8sjd" podStartSLOduration=2.359011907 podStartE2EDuration="4.026630752s" totalImagesPullingTime="1.667618845s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:41:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:41:58.025643521 +0000 UTC m=+56.982220989" watchObservedRunningTime="2026-07-07 15:41:58.026630752 +0000 UTC m=+56.983208218" Jul 07 15:41:58.098703 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:58.098668 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:41:58.194828 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:58.194800 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:41:58.599797 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:58.599766 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:41:59.279673 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:59.279641 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:41:59.531096 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:59.531010 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:41:59.672396 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:59.672364 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:41:59.774696 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:59.774663 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:41:59.875312 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:41:59.875281 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:42:00.130236 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:00.130137 2565 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-68b757fb67-s9kl8" Jul 07 15:42:00.130236 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:00.130196 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-68b757fb67-s9kl8" Jul 07 15:42:00.130539 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:00.130518 2565 scope.go:117] "RemoveContainer" containerID="ac7cb63b1b61c1458da32332e3b26404b52afe757964b7dc3c5fd41e55bae8fa" Jul 07 15:42:00.130680 ip-10-0-132-65 kubenswrapper[2565]: E0707 15:42:00.130662 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-68b757fb67-s9kl8_openshift-console-operator(62da0165-d32c-4bba-9844-a49154d765e5)\"" pod="openshift-console-operator/console-operator-68b757fb67-s9kl8" podUID="62da0165-d32c-4bba-9844-a49154d765e5" Jul 07 15:42:00.157438 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:00.157411 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:42:00.568504 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:00.568459 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:42:00.723413 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:00.723381 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/12101a9b-a592-4514-9eaf-513f0feeb2e9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vthmh\" (UID: \"12101a9b-a592-4514-9eaf-513f0feeb2e9\") " pod="openshift-insights/insights-runtime-extractor-vthmh" Jul 07 15:42:00.723619 ip-10-0-132-65 kubenswrapper[2565]: E0707 15:42:00.723580 2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 07 15:42:00.723686 ip-10-0-132-65 kubenswrapper[2565]: E0707 15:42:00.723660 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/12101a9b-a592-4514-9eaf-513f0feeb2e9-insights-runtime-extractor-tls podName:12101a9b-a592-4514-9eaf-513f0feeb2e9 nodeName:}" failed. No retries permitted until 2026-07-07 15:42:16.723637553 +0000 UTC m=+75.680215012 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/12101a9b-a592-4514-9eaf-513f0feeb2e9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vthmh" (UID: "12101a9b-a592-4514-9eaf-513f0feeb2e9") : secret "insights-runtime-extractor-tls" not found Jul 07 15:42:01.675444 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:01.675412 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:42:01.821823 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:01.821792 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:42:01.954976 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:01.954894 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:42:02.480084 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:02.480048 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:42:02.580641 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:02.580608 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:42:02.826508 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:02.826474 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:42:02.926367 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:02.926337 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-942xh" Jul 07 15:42:02.952117 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:02.952084 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:42:03.274720 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:03.274651 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:42:03.376041 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:03.376005 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:42:03.767316 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:03.767277 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:42:04.256313 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:04.256278 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:42:04.357562 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:04.357525 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:42:04.544472 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:04.544388 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:42:05.163299 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:05.163268 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:42:05.747939 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:05.747909 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:42:05.945032 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:05.945003 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:42:06.350862 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:06.350831 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:42:06.462163 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:06.462124 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:42:06.470233 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:06.470208 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0c5071d6-cead-4992-a7e9-9a752b955cae-original-pull-secret\") pod \"global-pull-secret-syncer-6qzkn\" (UID: \"0c5071d6-cead-4992-a7e9-9a752b955cae\") " pod="kube-system/global-pull-secret-syncer-6qzkn" Jul 07 15:42:06.470313 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:06.470251 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/301c1027-169c-47f5-a3ea-89cde78790d2-metrics-certs\") pod \"network-metrics-daemon-sdzxq\" (UID: \"301c1027-169c-47f5-a3ea-89cde78790d2\") " pod="openshift-multus/network-metrics-daemon-sdzxq" Jul 07 15:42:06.473155 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:06.473135 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 07 15:42:06.473155 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:06.473144 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 07 15:42:06.480560 ip-10-0-132-65 kubenswrapper[2565]: E0707 15:42:06.480539 2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 07 15:42:06.480618 ip-10-0-132-65 kubenswrapper[2565]: E0707 15:42:06.480607 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/301c1027-169c-47f5-a3ea-89cde78790d2-metrics-certs podName:301c1027-169c-47f5-a3ea-89cde78790d2 nodeName:}" failed. No retries permitted until 2026-07-07 15:43:10.480588935 +0000 UTC m=+129.437166380 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/301c1027-169c-47f5-a3ea-89cde78790d2-metrics-certs") pod "network-metrics-daemon-sdzxq" (UID: "301c1027-169c-47f5-a3ea-89cde78790d2") : secret "metrics-daemon-secret" not found Jul 07 15:42:06.482750 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:06.482730 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0c5071d6-cead-4992-a7e9-9a752b955cae-original-pull-secret\") pod \"global-pull-secret-syncer-6qzkn\" (UID: \"0c5071d6-cead-4992-a7e9-9a752b955cae\") " pod="kube-system/global-pull-secret-syncer-6qzkn" Jul 07 15:42:06.557678 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:06.557651 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:42:06.564718 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:06.564700 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6qzkn" Jul 07 15:42:06.571225 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:06.571199 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6brw\" (UniqueName: \"kubernetes.io/projected/e1f2bb91-99ac-4353-b790-622cb6d40663-kube-api-access-n6brw\") pod \"network-check-target-94vlx\" (UID: \"e1f2bb91-99ac-4353-b790-622cb6d40663\") " pod="openshift-network-diagnostics/network-check-target-94vlx" Jul 07 15:42:06.574407 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:06.574383 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 07 15:42:06.584809 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:06.584788 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 07 15:42:06.594226 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:06.594205 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6brw\" (UniqueName: \"kubernetes.io/projected/e1f2bb91-99ac-4353-b790-622cb6d40663-kube-api-access-n6brw\") pod \"network-check-target-94vlx\" (UID: \"e1f2bb91-99ac-4353-b790-622cb6d40663\") " pod="openshift-network-diagnostics/network-check-target-94vlx" Jul 07 15:42:06.689039 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:06.689010 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-6qzkn"] Jul 07 15:42:06.692297 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:42:06.692265 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c5071d6_cead_4992_a7e9_9a752b955cae.slice/crio-eabf5e02db3fc58d9148887ae871658d6e74d764c91e56315786739d1fe7c7fb WatchSource:0}: Error finding container eabf5e02db3fc58d9148887ae871658d6e74d764c91e56315786739d1fe7c7fb: Status 404 returned error can't find the container with id eabf5e02db3fc58d9148887ae871658d6e74d764c91e56315786739d1fe7c7fb Jul 07 15:42:06.878849 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:06.878770 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-k55hz\"" Jul 07 15:42:06.886367 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:06.886346 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-94vlx" Jul 07 15:42:06.953271 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:06.953242 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:42:07.009952 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:07.009923 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-94vlx"] Jul 07 15:42:07.012870 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:42:07.012845 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1f2bb91_99ac_4353_b790_622cb6d40663.slice/crio-4ae36dd0c3c50c05012fe1a0ef4b9e5d3ee547f175f05bed84e14d91bdfecfa9 WatchSource:0}: Error finding container 4ae36dd0c3c50c05012fe1a0ef4b9e5d3ee547f175f05bed84e14d91bdfecfa9: Status 404 returned error can't find the container with id 4ae36dd0c3c50c05012fe1a0ef4b9e5d3ee547f175f05bed84e14d91bdfecfa9 Jul 07 15:42:07.034830 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:07.034800 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-94vlx" event={"ID":"e1f2bb91-99ac-4353-b790-622cb6d40663","Type":"ContainerStarted","Data":"4ae36dd0c3c50c05012fe1a0ef4b9e5d3ee547f175f05bed84e14d91bdfecfa9"} Jul 07 15:42:07.035867 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:07.035839 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-6qzkn" event={"ID":"0c5071d6-cead-4992-a7e9-9a752b955cae","Type":"ContainerStarted","Data":"eabf5e02db3fc58d9148887ae871658d6e74d764c91e56315786739d1fe7c7fb"} Jul 07 15:42:07.551279 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:07.551243 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:42:08.153116 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:08.153078 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:42:08.344171 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:08.344139 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:42:08.548637 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:08.548552 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:42:08.791820 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:08.791784 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cb4ddf17-3bbe-4d75-9510-99f4a61da167-metrics-tls\") pod \"dns-default-qhqsf\" (UID: \"cb4ddf17-3bbe-4d75-9510-99f4a61da167\") " pod="openshift-dns/dns-default-qhqsf" Jul 07 15:42:08.794750 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:08.794719 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/cb4ddf17-3bbe-4d75-9510-99f4a61da167-metrics-tls\") pod \"dns-default-qhqsf\" (UID: \"cb4ddf17-3bbe-4d75-9510-99f4a61da167\") " pod="openshift-dns/dns-default-qhqsf" Jul 07 15:42:08.953998 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:08.953964 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:42:09.017440 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:09.017408 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-w75vl\"" Jul 07 15:42:09.025634 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:09.025513 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-qhqsf" Jul 07 15:42:09.189332 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:09.189297 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-qhqsf"] Jul 07 15:42:09.547868 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:09.547834 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:42:09.683888 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:09.683829 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:42:09.766962 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:09.766930 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:42:09.952102 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:09.952065 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:42:10.214579 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:42:10.214513 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb4ddf17_3bbe_4d75_9510_99f4a61da167.slice/crio-2a917f74679a351b6488fd2aaf7f21f98051d15ff5b74af9c37032173254afcd WatchSource:0}: Error finding container 2a917f74679a351b6488fd2aaf7f21f98051d15ff5b74af9c37032173254afcd: Status 404 returned error can't find the container with id 2a917f74679a351b6488fd2aaf7f21f98051d15ff5b74af9c37032173254afcd Jul 07 15:42:10.438799 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:10.438764 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:42:11.052850 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:11.052808 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-qhqsf" event={"ID":"cb4ddf17-3bbe-4d75-9510-99f4a61da167","Type":"ContainerStarted","Data":"2a917f74679a351b6488fd2aaf7f21f98051d15ff5b74af9c37032173254afcd"} Jul 07 15:42:11.511930 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:11.511884 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/95b55bd6-e205-4969-ac44-d3f67f63f746-metrics-certs\") pod \"router-default-6b6cdf54ff-tjwbr\" (UID: \"95b55bd6-e205-4969-ac44-d3f67f63f746\") " pod="openshift-ingress/router-default-6b6cdf54ff-tjwbr" Jul 07 15:42:11.511930 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:11.511941 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/95b55bd6-e205-4969-ac44-d3f67f63f746-service-ca-bundle\") pod \"router-default-6b6cdf54ff-tjwbr\" (UID: \"95b55bd6-e205-4969-ac44-d3f67f63f746\") " pod="openshift-ingress/router-default-6b6cdf54ff-tjwbr" Jul 07 15:42:11.512688 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:11.512665 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/95b55bd6-e205-4969-ac44-d3f67f63f746-service-ca-bundle\") pod \"router-default-6b6cdf54ff-tjwbr\" (UID: \"95b55bd6-e205-4969-ac44-d3f67f63f746\") " pod="openshift-ingress/router-default-6b6cdf54ff-tjwbr" Jul 07 15:42:11.514648 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:11.514627 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/95b55bd6-e205-4969-ac44-d3f67f63f746-metrics-certs\") pod \"router-default-6b6cdf54ff-tjwbr\" (UID: \"95b55bd6-e205-4969-ac44-d3f67f63f746\") " pod="openshift-ingress/router-default-6b6cdf54ff-tjwbr" Jul 07 15:42:11.786845 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:11.786774 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-2cmdl\"" Jul 07 15:42:11.794346 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:11.794323 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-6b6cdf54ff-tjwbr" Jul 07 15:42:12.150303 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:12.150272 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-6b6cdf54ff-tjwbr"] Jul 07 15:42:12.153400 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:42:12.153361 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95b55bd6_e205_4969_ac44_d3f67f63f746.slice/crio-3558d87f42700c9c5d863493116592b3c623d2f52009937c19485ce325dce13a WatchSource:0}: Error finding container 3558d87f42700c9c5d863493116592b3c623d2f52009937c19485ce325dce13a: Status 404 returned error can't find the container with id 3558d87f42700c9c5d863493116592b3c623d2f52009937c19485ce325dce13a Jul 07 15:42:13.064203 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:13.064161 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-94vlx" event={"ID":"e1f2bb91-99ac-4353-b790-622cb6d40663","Type":"ContainerStarted","Data":"9684f1e9d3b537a3672508f1402a4916a5631bab1efd8e397bdfd9e58720cde3"} Jul 07 15:42:13.064748 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:13.064292 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-94vlx" Jul 07 15:42:13.066361 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:13.066271 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-6qzkn" event={"ID":"0c5071d6-cead-4992-a7e9-9a752b955cae","Type":"ContainerStarted","Data":"b72b17f2107e8768c961e8a08395c0e7a2dbee46df7aaa9221de885bf7135f77"} Jul 07 15:42:13.068037 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:13.068011 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-6b6cdf54ff-tjwbr" event={"ID":"95b55bd6-e205-4969-ac44-d3f67f63f746","Type":"ContainerStarted","Data":"779084df11f5da049896f0ff2d4e120a2fa59cd4a4252025decc2fd59f334644"} Jul 07 15:42:13.068159 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:13.068048 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-6b6cdf54ff-tjwbr" event={"ID":"95b55bd6-e205-4969-ac44-d3f67f63f746","Type":"ContainerStarted","Data":"3558d87f42700c9c5d863493116592b3c623d2f52009937c19485ce325dce13a"} Jul 07 15:42:13.083179 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:13.083126 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-94vlx" podStartSLOduration=67.101672479 podStartE2EDuration="1m12.083107628s" totalImagesPullingTime="4.981435149s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:41:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:42:13.08269739 +0000 UTC m=+72.039274856" watchObservedRunningTime="2026-07-07 15:42:13.083107628 +0000 UTC m=+72.039685095" Jul 07 15:42:13.103722 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:13.103666 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-6qzkn" podStartSLOduration=66.789865232 podStartE2EDuration="1m12.10364429s" totalImagesPullingTime="5.313779058s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 15:41:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:42:13.102745685 +0000 UTC m=+72.059323153" watchObservedRunningTime="2026-07-07 15:42:13.10364429 +0000 UTC m=+72.060221758" Jul 07 15:42:13.127233 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:13.127183 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-6b6cdf54ff-tjwbr" podStartSLOduration=34.127169089 podStartE2EDuration="34.127169089s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:41:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:42:13.125877706 +0000 UTC m=+72.082455175" watchObservedRunningTime="2026-07-07 15:42:13.127169089 +0000 UTC m=+72.083746556" Jul 07 15:42:13.655478 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:13.655441 2565 scope.go:117] "RemoveContainer" containerID="ac7cb63b1b61c1458da32332e3b26404b52afe757964b7dc3c5fd41e55bae8fa" Jul 07 15:42:13.795045 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:13.795006 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-6b6cdf54ff-tjwbr" Jul 07 15:42:13.797555 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:13.797529 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-6b6cdf54ff-tjwbr" Jul 07 15:42:14.073141 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:14.073118 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-s9kl8_62da0165-d32c-4bba-9844-a49154d765e5/console-operator/1.log" Jul 07 15:42:14.073623 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:14.073219 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-68b757fb67-s9kl8" event={"ID":"62da0165-d32c-4bba-9844-a49154d765e5","Type":"ContainerStarted","Data":"25d189d1605e8727661965657b5945ecc51e84726bbfa2bede5f489b47b0eb4a"} Jul 07 15:42:14.073623 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:14.073526 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-68b757fb67-s9kl8" Jul 07 15:42:14.075141 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:14.075119 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-qhqsf" event={"ID":"cb4ddf17-3bbe-4d75-9510-99f4a61da167","Type":"ContainerStarted","Data":"29a91e12cd442b7dad40ce2f01007623cfb1d8e3fa943a9e8106e27382e46b2b"} Jul 07 15:42:14.075266 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:14.075172 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-6b6cdf54ff-tjwbr" Jul 07 15:42:14.075531 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:14.075518 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-qhqsf" Jul 07 15:42:14.075616 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:14.075555 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-qhqsf" event={"ID":"cb4ddf17-3bbe-4d75-9510-99f4a61da167","Type":"ContainerStarted","Data":"1c035b35c2e71026539ac666aafbeab4ceea964a912f5a6c5203d2f429398e4e"} Jul 07 15:42:14.076421 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:14.076404 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-6b6cdf54ff-tjwbr" Jul 07 15:42:14.093113 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:14.093074 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-68b757fb67-s9kl8" podStartSLOduration=22.10568403 podStartE2EDuration="25.09305459s" totalImagesPullingTime="2.98737056s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:41:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:42:14.092564483 +0000 UTC m=+73.049141950" watchObservedRunningTime="2026-07-07 15:42:14.09305459 +0000 UTC m=+73.049632095" Jul 07 15:42:14.112676 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:14.112611 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-qhqsf" podStartSLOduration=35.185178255 podStartE2EDuration="38.112595009s" totalImagesPullingTime="2.927416754s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:41:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:42:14.112210998 +0000 UTC m=+73.068788465" watchObservedRunningTime="2026-07-07 15:42:14.112595009 +0000 UTC m=+73.069172478" Jul 07 15:42:14.587507 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:14.587460 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-68b757fb67-s9kl8" Jul 07 15:42:14.787634 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:14.787600 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:42:14.892562 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:14.892461 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:42:14.994642 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:14.994611 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:42:15.097688 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:15.097658 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:42:15.195126 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:15.195041 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:42:15.295458 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:15.295425 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:42:15.464588 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:15.464483 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:42:15.576836 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:15.576804 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:42:15.864766 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:15.864738 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:42:15.964916 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:15.964887 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:42:16.679314 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:16.679282 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:42:16.752219 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:16.752175 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/12101a9b-a592-4514-9eaf-513f0feeb2e9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vthmh\" (UID: \"12101a9b-a592-4514-9eaf-513f0feeb2e9\") " pod="openshift-insights/insights-runtime-extractor-vthmh" Jul 07 15:42:16.754582 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:16.754561 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/12101a9b-a592-4514-9eaf-513f0feeb2e9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vthmh\" (UID: \"12101a9b-a592-4514-9eaf-513f0feeb2e9\") " pod="openshift-insights/insights-runtime-extractor-vthmh" Jul 07 15:42:16.874043 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:16.874013 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:42:16.957441 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:16.957363 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-4xbb9\"" Jul 07 15:42:16.965056 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:16.965035 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-vthmh" Jul 07 15:42:17.097786 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:17.097757 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-vthmh"] Jul 07 15:42:17.100905 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:42:17.100879 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12101a9b_a592_4514_9eaf_513f0feeb2e9.slice/crio-36a8bbdecd2a2c068c6947cc406286f072e7a9afdf35eaf97b1cf677e240cdc2 WatchSource:0}: Error finding container 36a8bbdecd2a2c068c6947cc406286f072e7a9afdf35eaf97b1cf677e240cdc2: Status 404 returned error can't find the container with id 36a8bbdecd2a2c068c6947cc406286f072e7a9afdf35eaf97b1cf677e240cdc2 Jul 07 15:42:17.313231 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:17.313202 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:42:17.394142 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:17.394087 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:42:17.543048 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:17.542967 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:42:18.089335 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:18.089297 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vthmh" event={"ID":"12101a9b-a592-4514-9eaf-513f0feeb2e9","Type":"ContainerStarted","Data":"36a8bbdecd2a2c068c6947cc406286f072e7a9afdf35eaf97b1cf677e240cdc2"} Jul 07 15:42:19.122578 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:19.122538 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-6bffdf9778-wz5k7"] Jul 07 15:42:19.130328 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:19.130303 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6bffdf9778-wz5k7" Jul 07 15:42:19.145169 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:19.145138 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-xns5k\"" Jul 07 15:42:19.146978 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:19.146952 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Jul 07 15:42:19.147125 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:19.147108 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Jul 07 15:42:19.153810 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:19.153785 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6d555fc9d8-q7czh"] Jul 07 15:42:19.157013 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:19.156994 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6d555fc9d8-q7czh" Jul 07 15:42:19.163474 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:19.162918 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 07 15:42:19.163474 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:19.163043 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\"" Jul 07 15:42:19.163474 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:19.163226 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\"" Jul 07 15:42:19.164244 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:19.164219 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\"" Jul 07 15:42:19.164351 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:19.164269 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 07 15:42:19.164465 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:19.164351 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\"" Jul 07 15:42:19.164465 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:19.164432 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 07 15:42:19.168324 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:19.168301 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6bffdf9778-wz5k7"] Jul 07 15:42:19.171409 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:19.171380 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0cbd6022-2c42-43b2-950f-ab089797a659-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-wz5k7\" (UID: \"0cbd6022-2c42-43b2-950f-ab089797a659\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-wz5k7" Jul 07 15:42:19.171526 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:19.171453 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/0cbd6022-2c42-43b2-950f-ab089797a659-nginx-conf\") pod \"networking-console-plugin-6bffdf9778-wz5k7\" (UID: \"0cbd6022-2c42-43b2-950f-ab089797a659\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-wz5k7" Jul 07 15:42:19.182430 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:19.182400 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6d555fc9d8-q7czh"] Jul 07 15:42:19.241839 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:19.241810 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7f7d849b7d-bvxw4"] Jul 07 15:42:19.246209 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:19.246177 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7f7d849b7d-bvxw4" Jul 07 15:42:19.249671 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:19.249645 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 07 15:42:19.249851 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:19.249736 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 07 15:42:19.249935 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:19.249876 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-bxv9m\"" Jul 07 15:42:19.249994 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:19.249967 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 07 15:42:19.255142 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:19.255107 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 07 15:42:19.272029 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:19.272001 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7f7d849b7d-bvxw4"] Jul 07 15:42:19.272172 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:19.272150 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/0cbd6022-2c42-43b2-950f-ab089797a659-nginx-conf\") pod \"networking-console-plugin-6bffdf9778-wz5k7\" (UID: \"0cbd6022-2c42-43b2-950f-ab089797a659\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-wz5k7" Jul 07 15:42:19.272236 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:19.272186 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/60091952-ccf9-4f73-a63d-534acfcfb1ae-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6d555fc9d8-q7czh\" (UID: \"60091952-ccf9-4f73-a63d-534acfcfb1ae\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6d555fc9d8-q7czh" Jul 07 15:42:19.272289 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:19.272238 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/60091952-ccf9-4f73-a63d-534acfcfb1ae-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6d555fc9d8-q7czh\" (UID: \"60091952-ccf9-4f73-a63d-534acfcfb1ae\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6d555fc9d8-q7czh" Jul 07 15:42:19.272289 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:19.272268 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/60091952-ccf9-4f73-a63d-534acfcfb1ae-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6d555fc9d8-q7czh\" (UID: \"60091952-ccf9-4f73-a63d-534acfcfb1ae\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6d555fc9d8-q7czh" Jul 07 15:42:19.272392 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:19.272312 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/60091952-ccf9-4f73-a63d-534acfcfb1ae-ca\") pod \"cluster-proxy-proxy-agent-6d555fc9d8-q7czh\" (UID: \"60091952-ccf9-4f73-a63d-534acfcfb1ae\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6d555fc9d8-q7czh" Jul 07 15:42:19.272392 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:19.272344 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwsbw\" (UniqueName: \"kubernetes.io/projected/60091952-ccf9-4f73-a63d-534acfcfb1ae-kube-api-access-hwsbw\") pod \"cluster-proxy-proxy-agent-6d555fc9d8-q7czh\" (UID: \"60091952-ccf9-4f73-a63d-534acfcfb1ae\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6d555fc9d8-q7czh" Jul 07 15:42:19.272392 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:19.272386 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0cbd6022-2c42-43b2-950f-ab089797a659-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-wz5k7\" (UID: \"0cbd6022-2c42-43b2-950f-ab089797a659\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-wz5k7" Jul 07 15:42:19.272565 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:19.272415 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/60091952-ccf9-4f73-a63d-534acfcfb1ae-hub\") pod \"cluster-proxy-proxy-agent-6d555fc9d8-q7czh\" (UID: \"60091952-ccf9-4f73-a63d-534acfcfb1ae\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6d555fc9d8-q7czh" Jul 07 15:42:19.272877 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:19.272860 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/0cbd6022-2c42-43b2-950f-ab089797a659-nginx-conf\") pod \"networking-console-plugin-6bffdf9778-wz5k7\" (UID: \"0cbd6022-2c42-43b2-950f-ab089797a659\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-wz5k7" Jul 07 15:42:19.275098 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:19.275075 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0cbd6022-2c42-43b2-950f-ab089797a659-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-wz5k7\" (UID: \"0cbd6022-2c42-43b2-950f-ab089797a659\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-wz5k7" Jul 07 15:42:19.373382 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:19.373352 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/60091952-ccf9-4f73-a63d-534acfcfb1ae-hub\") pod \"cluster-proxy-proxy-agent-6d555fc9d8-q7czh\" (UID: \"60091952-ccf9-4f73-a63d-534acfcfb1ae\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6d555fc9d8-q7czh" Jul 07 15:42:19.373382 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:19.373389 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f9bae27f-4189-4ec5-9f1f-b2036270ebe3-registry-tls\") pod \"image-registry-7f7d849b7d-bvxw4\" (UID: \"f9bae27f-4189-4ec5-9f1f-b2036270ebe3\") " pod="openshift-image-registry/image-registry-7f7d849b7d-bvxw4" Jul 07 15:42:19.373628 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:19.373447 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f9bae27f-4189-4ec5-9f1f-b2036270ebe3-trusted-ca\") pod \"image-registry-7f7d849b7d-bvxw4\" (UID: \"f9bae27f-4189-4ec5-9f1f-b2036270ebe3\") " pod="openshift-image-registry/image-registry-7f7d849b7d-bvxw4" Jul 07 15:42:19.373628 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:19.373514 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f9bae27f-4189-4ec5-9f1f-b2036270ebe3-bound-sa-token\") pod \"image-registry-7f7d849b7d-bvxw4\" (UID: \"f9bae27f-4189-4ec5-9f1f-b2036270ebe3\") " pod="openshift-image-registry/image-registry-7f7d849b7d-bvxw4" Jul 07 15:42:19.373628 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:19.373541 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f9bae27f-4189-4ec5-9f1f-b2036270ebe3-installation-pull-secrets\") pod \"image-registry-7f7d849b7d-bvxw4\" (UID: \"f9bae27f-4189-4ec5-9f1f-b2036270ebe3\") " pod="openshift-image-registry/image-registry-7f7d849b7d-bvxw4" Jul 07 15:42:19.373628 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:19.373574 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/60091952-ccf9-4f73-a63d-534acfcfb1ae-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6d555fc9d8-q7czh\" (UID: \"60091952-ccf9-4f73-a63d-534acfcfb1ae\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6d555fc9d8-q7czh" Jul 07 15:42:19.373628 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:19.373604 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djsrn\" (UniqueName: \"kubernetes.io/projected/f9bae27f-4189-4ec5-9f1f-b2036270ebe3-kube-api-access-djsrn\") pod \"image-registry-7f7d849b7d-bvxw4\" (UID: \"f9bae27f-4189-4ec5-9f1f-b2036270ebe3\") " pod="openshift-image-registry/image-registry-7f7d849b7d-bvxw4" Jul 07 15:42:19.373853 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:19.373632 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f9bae27f-4189-4ec5-9f1f-b2036270ebe3-ca-trust-extracted\") pod \"image-registry-7f7d849b7d-bvxw4\" (UID: \"f9bae27f-4189-4ec5-9f1f-b2036270ebe3\") " pod="openshift-image-registry/image-registry-7f7d849b7d-bvxw4" Jul 07 15:42:19.373853 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:19.373713 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/60091952-ccf9-4f73-a63d-534acfcfb1ae-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6d555fc9d8-q7czh\" (UID: \"60091952-ccf9-4f73-a63d-534acfcfb1ae\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6d555fc9d8-q7czh" Jul 07 15:42:19.373853 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:19.373740 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/60091952-ccf9-4f73-a63d-534acfcfb1ae-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6d555fc9d8-q7czh\" (UID: \"60091952-ccf9-4f73-a63d-534acfcfb1ae\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6d555fc9d8-q7czh" Jul 07 15:42:19.373853 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:19.373769 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/60091952-ccf9-4f73-a63d-534acfcfb1ae-ca\") pod \"cluster-proxy-proxy-agent-6d555fc9d8-q7czh\" (UID: \"60091952-ccf9-4f73-a63d-534acfcfb1ae\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6d555fc9d8-q7czh" Jul 07 15:42:19.373853 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:19.373801 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwsbw\" (UniqueName: \"kubernetes.io/projected/60091952-ccf9-4f73-a63d-534acfcfb1ae-kube-api-access-hwsbw\") pod \"cluster-proxy-proxy-agent-6d555fc9d8-q7czh\" (UID: \"60091952-ccf9-4f73-a63d-534acfcfb1ae\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6d555fc9d8-q7czh" Jul 07 15:42:19.373853 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:19.373827 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f9bae27f-4189-4ec5-9f1f-b2036270ebe3-registry-certificates\") pod \"image-registry-7f7d849b7d-bvxw4\" (UID: \"f9bae27f-4189-4ec5-9f1f-b2036270ebe3\") " pod="openshift-image-registry/image-registry-7f7d849b7d-bvxw4" Jul 07 15:42:19.374155 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:19.373868 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f9bae27f-4189-4ec5-9f1f-b2036270ebe3-image-registry-private-configuration\") pod \"image-registry-7f7d849b7d-bvxw4\" (UID: \"f9bae27f-4189-4ec5-9f1f-b2036270ebe3\") " pod="openshift-image-registry/image-registry-7f7d849b7d-bvxw4" Jul 07 15:42:19.374522 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:19.374457 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/60091952-ccf9-4f73-a63d-534acfcfb1ae-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6d555fc9d8-q7czh\" (UID: \"60091952-ccf9-4f73-a63d-534acfcfb1ae\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6d555fc9d8-q7czh" Jul 07 15:42:19.376624 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:19.376448 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/60091952-ccf9-4f73-a63d-534acfcfb1ae-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6d555fc9d8-q7czh\" (UID: \"60091952-ccf9-4f73-a63d-534acfcfb1ae\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6d555fc9d8-q7czh" Jul 07 15:42:19.376624 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:19.376578 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/60091952-ccf9-4f73-a63d-534acfcfb1ae-hub\") pod \"cluster-proxy-proxy-agent-6d555fc9d8-q7czh\" (UID: \"60091952-ccf9-4f73-a63d-534acfcfb1ae\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6d555fc9d8-q7czh" Jul 07 15:42:19.376624 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:19.376582 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/60091952-ccf9-4f73-a63d-534acfcfb1ae-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6d555fc9d8-q7czh\" (UID: \"60091952-ccf9-4f73-a63d-534acfcfb1ae\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6d555fc9d8-q7czh" Jul 07 15:42:19.376805 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:19.376782 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/60091952-ccf9-4f73-a63d-534acfcfb1ae-ca\") pod \"cluster-proxy-proxy-agent-6d555fc9d8-q7czh\" (UID: \"60091952-ccf9-4f73-a63d-534acfcfb1ae\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6d555fc9d8-q7czh" Jul 07 15:42:19.385932 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:19.385908 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwsbw\" (UniqueName: \"kubernetes.io/projected/60091952-ccf9-4f73-a63d-534acfcfb1ae-kube-api-access-hwsbw\") pod \"cluster-proxy-proxy-agent-6d555fc9d8-q7czh\" (UID: \"60091952-ccf9-4f73-a63d-534acfcfb1ae\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6d555fc9d8-q7czh" Jul 07 15:42:19.395853 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:19.395829 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:42:19.440837 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:19.440805 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6bffdf9778-wz5k7" Jul 07 15:42:19.474904 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:19.474862 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f9bae27f-4189-4ec5-9f1f-b2036270ebe3-registry-certificates\") pod \"image-registry-7f7d849b7d-bvxw4\" (UID: \"f9bae27f-4189-4ec5-9f1f-b2036270ebe3\") " pod="openshift-image-registry/image-registry-7f7d849b7d-bvxw4" Jul 07 15:42:19.475063 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:19.474923 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f9bae27f-4189-4ec5-9f1f-b2036270ebe3-image-registry-private-configuration\") pod \"image-registry-7f7d849b7d-bvxw4\" (UID: \"f9bae27f-4189-4ec5-9f1f-b2036270ebe3\") " pod="openshift-image-registry/image-registry-7f7d849b7d-bvxw4" Jul 07 15:42:19.475063 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:19.474972 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f9bae27f-4189-4ec5-9f1f-b2036270ebe3-registry-tls\") pod \"image-registry-7f7d849b7d-bvxw4\" (UID: \"f9bae27f-4189-4ec5-9f1f-b2036270ebe3\") " pod="openshift-image-registry/image-registry-7f7d849b7d-bvxw4" Jul 07 15:42:19.475063 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:19.474991 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f9bae27f-4189-4ec5-9f1f-b2036270ebe3-trusted-ca\") pod \"image-registry-7f7d849b7d-bvxw4\" (UID: \"f9bae27f-4189-4ec5-9f1f-b2036270ebe3\") " pod="openshift-image-registry/image-registry-7f7d849b7d-bvxw4" Jul 07 15:42:19.475063 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:19.475033 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f9bae27f-4189-4ec5-9f1f-b2036270ebe3-bound-sa-token\") pod \"image-registry-7f7d849b7d-bvxw4\" (UID: \"f9bae27f-4189-4ec5-9f1f-b2036270ebe3\") " pod="openshift-image-registry/image-registry-7f7d849b7d-bvxw4" Jul 07 15:42:19.475282 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:19.475177 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f9bae27f-4189-4ec5-9f1f-b2036270ebe3-installation-pull-secrets\") pod \"image-registry-7f7d849b7d-bvxw4\" (UID: \"f9bae27f-4189-4ec5-9f1f-b2036270ebe3\") " pod="openshift-image-registry/image-registry-7f7d849b7d-bvxw4" Jul 07 15:42:19.475282 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:19.475238 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djsrn\" (UniqueName: \"kubernetes.io/projected/f9bae27f-4189-4ec5-9f1f-b2036270ebe3-kube-api-access-djsrn\") pod \"image-registry-7f7d849b7d-bvxw4\" (UID: \"f9bae27f-4189-4ec5-9f1f-b2036270ebe3\") " pod="openshift-image-registry/image-registry-7f7d849b7d-bvxw4" Jul 07 15:42:19.475282 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:19.475275 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f9bae27f-4189-4ec5-9f1f-b2036270ebe3-ca-trust-extracted\") pod \"image-registry-7f7d849b7d-bvxw4\" (UID: \"f9bae27f-4189-4ec5-9f1f-b2036270ebe3\") " pod="openshift-image-registry/image-registry-7f7d849b7d-bvxw4" Jul 07 15:42:19.475720 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:19.475686 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f9bae27f-4189-4ec5-9f1f-b2036270ebe3-ca-trust-extracted\") pod \"image-registry-7f7d849b7d-bvxw4\" (UID: \"f9bae27f-4189-4ec5-9f1f-b2036270ebe3\") " pod="openshift-image-registry/image-registry-7f7d849b7d-bvxw4" Jul 07 15:42:19.475841 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:19.475816 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f9bae27f-4189-4ec5-9f1f-b2036270ebe3-registry-certificates\") pod \"image-registry-7f7d849b7d-bvxw4\" (UID: \"f9bae27f-4189-4ec5-9f1f-b2036270ebe3\") " pod="openshift-image-registry/image-registry-7f7d849b7d-bvxw4" Jul 07 15:42:19.475951 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:19.475930 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f9bae27f-4189-4ec5-9f1f-b2036270ebe3-trusted-ca\") pod \"image-registry-7f7d849b7d-bvxw4\" (UID: \"f9bae27f-4189-4ec5-9f1f-b2036270ebe3\") " pod="openshift-image-registry/image-registry-7f7d849b7d-bvxw4" Jul 07 15:42:19.476020 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:19.475952 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6d555fc9d8-q7czh" Jul 07 15:42:19.477657 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:19.477637 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f9bae27f-4189-4ec5-9f1f-b2036270ebe3-image-registry-private-configuration\") pod \"image-registry-7f7d849b7d-bvxw4\" (UID: \"f9bae27f-4189-4ec5-9f1f-b2036270ebe3\") " pod="openshift-image-registry/image-registry-7f7d849b7d-bvxw4" Jul 07 15:42:19.478151 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:19.478127 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f9bae27f-4189-4ec5-9f1f-b2036270ebe3-registry-tls\") pod \"image-registry-7f7d849b7d-bvxw4\" (UID: \"f9bae27f-4189-4ec5-9f1f-b2036270ebe3\") " pod="openshift-image-registry/image-registry-7f7d849b7d-bvxw4" Jul 07 15:42:19.478421 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:19.478401 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f9bae27f-4189-4ec5-9f1f-b2036270ebe3-installation-pull-secrets\") pod \"image-registry-7f7d849b7d-bvxw4\" (UID: \"f9bae27f-4189-4ec5-9f1f-b2036270ebe3\") " pod="openshift-image-registry/image-registry-7f7d849b7d-bvxw4" Jul 07 15:42:19.485231 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:19.485207 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f9bae27f-4189-4ec5-9f1f-b2036270ebe3-bound-sa-token\") pod \"image-registry-7f7d849b7d-bvxw4\" (UID: \"f9bae27f-4189-4ec5-9f1f-b2036270ebe3\") " pod="openshift-image-registry/image-registry-7f7d849b7d-bvxw4" Jul 07 15:42:19.486371 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:19.486347 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-djsrn\" (UniqueName: \"kubernetes.io/projected/f9bae27f-4189-4ec5-9f1f-b2036270ebe3-kube-api-access-djsrn\") pod \"image-registry-7f7d849b7d-bvxw4\" (UID: \"f9bae27f-4189-4ec5-9f1f-b2036270ebe3\") " pod="openshift-image-registry/image-registry-7f7d849b7d-bvxw4" Jul 07 15:42:19.501079 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:19.501021 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:42:19.558555 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:19.558521 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7f7d849b7d-bvxw4" Jul 07 15:42:19.601952 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:19.601902 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:42:20.872153 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:20.872078 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7f7d849b7d-bvxw4"] Jul 07 15:42:20.874342 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:42:20.874318 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9bae27f_4189_4ec5_9f1f_b2036270ebe3.slice/crio-b25ba91f3facc77f82e430f48909936874f526319b2565bfb0a86ae58c518d05 WatchSource:0}: Error finding container b25ba91f3facc77f82e430f48909936874f526319b2565bfb0a86ae58c518d05: Status 404 returned error can't find the container with id b25ba91f3facc77f82e430f48909936874f526319b2565bfb0a86ae58c518d05 Jul 07 15:42:21.090652 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:21.090608 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6bffdf9778-wz5k7"] Jul 07 15:42:21.093379 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:42:21.093348 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0cbd6022_2c42_43b2_950f_ab089797a659.slice/crio-e78afe102c0f1fe21dba562a5447f959f25078ab9acc24b7e5ff220a25e8a66f WatchSource:0}: Error finding container e78afe102c0f1fe21dba562a5447f959f25078ab9acc24b7e5ff220a25e8a66f: Status 404 returned error can't find the container with id e78afe102c0f1fe21dba562a5447f959f25078ab9acc24b7e5ff220a25e8a66f Jul 07 15:42:21.098217 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:21.098177 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6d555fc9d8-q7czh"] Jul 07 15:42:21.101953 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:21.101921 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vthmh" event={"ID":"12101a9b-a592-4514-9eaf-513f0feeb2e9","Type":"ContainerStarted","Data":"982f67d66a5ca18f1e274c7d520076688115a769709a4c22e3ecd1f0137abd85"} Jul 07 15:42:21.102834 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:42:21.102807 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod60091952_ccf9_4f73_a63d_534acfcfb1ae.slice/crio-56dd06bec705b5505cb47c22372a1be54b6bbc5f0b8d277aeaedc128d62ee44a WatchSource:0}: Error finding container 56dd06bec705b5505cb47c22372a1be54b6bbc5f0b8d277aeaedc128d62ee44a: Status 404 returned error can't find the container with id 56dd06bec705b5505cb47c22372a1be54b6bbc5f0b8d277aeaedc128d62ee44a Jul 07 15:42:21.103148 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:21.103115 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6bffdf9778-wz5k7" event={"ID":"0cbd6022-2c42-43b2-950f-ab089797a659","Type":"ContainerStarted","Data":"e78afe102c0f1fe21dba562a5447f959f25078ab9acc24b7e5ff220a25e8a66f"} Jul 07 15:42:21.104718 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:21.104686 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7f7d849b7d-bvxw4" event={"ID":"f9bae27f-4189-4ec5-9f1f-b2036270ebe3","Type":"ContainerStarted","Data":"622ef7b196bc25b591f539c1b190ded34115f9125594544162b0572175eaf079"} Jul 07 15:42:21.104782 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:21.104741 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7f7d849b7d-bvxw4" event={"ID":"f9bae27f-4189-4ec5-9f1f-b2036270ebe3","Type":"ContainerStarted","Data":"b25ba91f3facc77f82e430f48909936874f526319b2565bfb0a86ae58c518d05"} Jul 07 15:42:21.104871 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:21.104850 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7f7d849b7d-bvxw4" Jul 07 15:42:21.130117 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:21.130061 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-7f7d849b7d-bvxw4" podStartSLOduration=2.130042978 podStartE2EDuration="2.130042978s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:42:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:42:21.129777698 +0000 UTC m=+80.086355168" watchObservedRunningTime="2026-07-07 15:42:21.130042978 +0000 UTC m=+80.086620444" Jul 07 15:42:22.112606 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:22.112507 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vthmh" event={"ID":"12101a9b-a592-4514-9eaf-513f0feeb2e9","Type":"ContainerStarted","Data":"4b227891e1cfbc2c7f81d8bb393a0bdf72102e8bed4069f5e4a3988c25fcaeba"} Jul 07 15:42:22.115446 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:22.115413 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6d555fc9d8-q7czh" event={"ID":"60091952-ccf9-4f73-a63d-534acfcfb1ae","Type":"ContainerStarted","Data":"56dd06bec705b5505cb47c22372a1be54b6bbc5f0b8d277aeaedc128d62ee44a"} Jul 07 15:42:22.308121 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:22.308081 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:42:22.401141 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:22.401041 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:42:24.317316 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:24.317181 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:42:24.401931 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:24.401854 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:42:25.084124 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:25.084096 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-qhqsf" Jul 07 15:42:25.188880 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:25.188846 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:42:25.274225 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:25.274189 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:42:26.133446 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:26.133404 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vthmh" event={"ID":"12101a9b-a592-4514-9eaf-513f0feeb2e9","Type":"ContainerStarted","Data":"bb11a210d840915f28d59dbdc3d0d764fbbfb36e2565c8107fd3f0a7ec4258b6"} Jul 07 15:42:26.135400 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:26.135376 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6d555fc9d8-q7czh" event={"ID":"60091952-ccf9-4f73-a63d-534acfcfb1ae","Type":"ContainerStarted","Data":"230d57124a17b319bdab11b3cfba7a2e23fab971fed81a7d76d557e84d103e83"} Jul 07 15:42:26.135522 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:26.135413 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6d555fc9d8-q7czh" event={"ID":"60091952-ccf9-4f73-a63d-534acfcfb1ae","Type":"ContainerStarted","Data":"ca19de937047c709915e8b036f8a763e11018516727ab0a18610a2d6cccdaad8"} Jul 07 15:42:26.135522 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:26.135431 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6d555fc9d8-q7czh" event={"ID":"60091952-ccf9-4f73-a63d-534acfcfb1ae","Type":"ContainerStarted","Data":"456f6788499a1c5d250a0e45de4c7e4943206bd8f16e3b4545ae26979328fe25"} Jul 07 15:42:26.136767 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:26.136746 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6bffdf9778-wz5k7" event={"ID":"0cbd6022-2c42-43b2-950f-ab089797a659","Type":"ContainerStarted","Data":"9a909170d7e3ec4421c850800d7cf0607d30da918e8e2fab2f6f2da6c32ddd7e"} Jul 07 15:42:26.157642 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:26.157555 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-vthmh" podStartSLOduration=33.527904809 podStartE2EDuration="42.157540314s" totalImagesPullingTime="8.629635505s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 15:41:44 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:42:26.155547677 +0000 UTC m=+85.112125144" watchObservedRunningTime="2026-07-07 15:42:26.157540314 +0000 UTC m=+85.114117814" Jul 07 15:42:26.174591 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:26.174538 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-6bffdf9778-wz5k7" podStartSLOduration=2.371077423 podStartE2EDuration="7.174524935s" totalImagesPullingTime="4.803447512s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:42:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:42:26.174132445 +0000 UTC m=+85.130709912" watchObservedRunningTime="2026-07-07 15:42:26.174524935 +0000 UTC m=+85.131102399" Jul 07 15:42:26.202434 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:26.202378 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6d555fc9d8-q7czh" podStartSLOduration=2.388079189 podStartE2EDuration="7.202357892s" totalImagesPullingTime="4.814278703s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:42:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:42:26.201090598 +0000 UTC m=+85.157668051" watchObservedRunningTime="2026-07-07 15:42:26.202357892 +0000 UTC m=+85.158935360" Jul 07 15:42:26.340833 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:26.340801 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:42:26.438402 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:26.438309 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:42:28.097032 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:28.097001 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:42:28.207531 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:28.207477 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:42:28.299205 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:28.299179 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:42:28.455518 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:28.455417 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:42:29.839692 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:29.839655 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:42:29.941251 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:29.941223 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:42:30.114779 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:30.114692 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:42:30.359095 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:30.359058 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:42:31.212130 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:31.212098 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:42:31.304217 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:31.304183 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:42:31.497072 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:31.496974 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:42:31.581114 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:31.581076 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:42:31.988197 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:31.988162 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:42:32.064177 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:32.064139 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:42:32.729331 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:32.729292 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:42:34.373047 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:34.373016 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:42:34.602300 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:34.602265 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:42:34.671849 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:34.671771 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-69d5v"] Jul 07 15:42:34.674834 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:34.674807 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-69d5v" Jul 07 15:42:34.677255 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:34.677230 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-p7tzj\"" Jul 07 15:42:34.677582 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:34.677563 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 07 15:42:34.677660 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:34.677598 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 07 15:42:34.678658 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:34.678641 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 07 15:42:34.678658 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:34.678647 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 07 15:42:34.678799 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:34.678654 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 07 15:42:34.678799 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:34.678679 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 07 15:42:34.721134 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:34.721108 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:42:34.799666 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:34.799631 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/767860ff-f41a-4b9b-962c-f0083d0f069a-node-exporter-textfile\") pod \"node-exporter-69d5v\" (UID: \"767860ff-f41a-4b9b-962c-f0083d0f069a\") " pod="openshift-monitoring/node-exporter-69d5v" Jul 07 15:42:34.799816 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:34.799682 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/767860ff-f41a-4b9b-962c-f0083d0f069a-node-exporter-accelerators-collector-config\") pod \"node-exporter-69d5v\" (UID: \"767860ff-f41a-4b9b-962c-f0083d0f069a\") " pod="openshift-monitoring/node-exporter-69d5v" Jul 07 15:42:34.799816 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:34.799724 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/767860ff-f41a-4b9b-962c-f0083d0f069a-metrics-client-ca\") pod \"node-exporter-69d5v\" (UID: \"767860ff-f41a-4b9b-962c-f0083d0f069a\") " pod="openshift-monitoring/node-exporter-69d5v" Jul 07 15:42:34.799816 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:34.799741 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/767860ff-f41a-4b9b-962c-f0083d0f069a-sys\") pod \"node-exporter-69d5v\" (UID: \"767860ff-f41a-4b9b-962c-f0083d0f069a\") " pod="openshift-monitoring/node-exporter-69d5v" Jul 07 15:42:34.799816 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:34.799758 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82nzg\" (UniqueName: \"kubernetes.io/projected/767860ff-f41a-4b9b-962c-f0083d0f069a-kube-api-access-82nzg\") pod \"node-exporter-69d5v\" (UID: \"767860ff-f41a-4b9b-962c-f0083d0f069a\") " pod="openshift-monitoring/node-exporter-69d5v" Jul 07 15:42:34.799962 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:34.799834 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/767860ff-f41a-4b9b-962c-f0083d0f069a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-69d5v\" (UID: \"767860ff-f41a-4b9b-962c-f0083d0f069a\") " pod="openshift-monitoring/node-exporter-69d5v" Jul 07 15:42:34.799962 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:34.799879 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/767860ff-f41a-4b9b-962c-f0083d0f069a-node-exporter-wtmp\") pod \"node-exporter-69d5v\" (UID: \"767860ff-f41a-4b9b-962c-f0083d0f069a\") " pod="openshift-monitoring/node-exporter-69d5v" Jul 07 15:42:34.799962 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:34.799902 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/767860ff-f41a-4b9b-962c-f0083d0f069a-node-exporter-tls\") pod \"node-exporter-69d5v\" (UID: \"767860ff-f41a-4b9b-962c-f0083d0f069a\") " pod="openshift-monitoring/node-exporter-69d5v" Jul 07 15:42:34.799962 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:34.799918 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/767860ff-f41a-4b9b-962c-f0083d0f069a-root\") pod \"node-exporter-69d5v\" (UID: \"767860ff-f41a-4b9b-962c-f0083d0f069a\") " pod="openshift-monitoring/node-exporter-69d5v" Jul 07 15:42:34.819374 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:34.819336 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:42:34.900431 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:34.900398 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/767860ff-f41a-4b9b-962c-f0083d0f069a-node-exporter-wtmp\") pod \"node-exporter-69d5v\" (UID: \"767860ff-f41a-4b9b-962c-f0083d0f069a\") " pod="openshift-monitoring/node-exporter-69d5v" Jul 07 15:42:34.900431 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:34.900435 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/767860ff-f41a-4b9b-962c-f0083d0f069a-node-exporter-tls\") pod \"node-exporter-69d5v\" (UID: \"767860ff-f41a-4b9b-962c-f0083d0f069a\") " pod="openshift-monitoring/node-exporter-69d5v" Jul 07 15:42:34.900676 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:34.900453 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/767860ff-f41a-4b9b-962c-f0083d0f069a-root\") pod \"node-exporter-69d5v\" (UID: \"767860ff-f41a-4b9b-962c-f0083d0f069a\") " pod="openshift-monitoring/node-exporter-69d5v" Jul 07 15:42:34.900676 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:34.900566 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/767860ff-f41a-4b9b-962c-f0083d0f069a-node-exporter-textfile\") pod \"node-exporter-69d5v\" (UID: \"767860ff-f41a-4b9b-962c-f0083d0f069a\") " pod="openshift-monitoring/node-exporter-69d5v" Jul 07 15:42:34.900676 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:34.900603 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/767860ff-f41a-4b9b-962c-f0083d0f069a-node-exporter-wtmp\") pod \"node-exporter-69d5v\" (UID: \"767860ff-f41a-4b9b-962c-f0083d0f069a\") " pod="openshift-monitoring/node-exporter-69d5v" Jul 07 15:42:34.900676 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:34.900627 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/767860ff-f41a-4b9b-962c-f0083d0f069a-node-exporter-accelerators-collector-config\") pod \"node-exporter-69d5v\" (UID: \"767860ff-f41a-4b9b-962c-f0083d0f069a\") " pod="openshift-monitoring/node-exporter-69d5v" Jul 07 15:42:34.900676 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:34.900603 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/767860ff-f41a-4b9b-962c-f0083d0f069a-root\") pod \"node-exporter-69d5v\" (UID: \"767860ff-f41a-4b9b-962c-f0083d0f069a\") " pod="openshift-monitoring/node-exporter-69d5v" Jul 07 15:42:34.900676 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:34.900662 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/767860ff-f41a-4b9b-962c-f0083d0f069a-metrics-client-ca\") pod \"node-exporter-69d5v\" (UID: \"767860ff-f41a-4b9b-962c-f0083d0f069a\") " pod="openshift-monitoring/node-exporter-69d5v" Jul 07 15:42:34.900923 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:34.900682 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/767860ff-f41a-4b9b-962c-f0083d0f069a-sys\") pod \"node-exporter-69d5v\" (UID: \"767860ff-f41a-4b9b-962c-f0083d0f069a\") " pod="openshift-monitoring/node-exporter-69d5v" Jul 07 15:42:34.900923 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:34.900702 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82nzg\" (UniqueName: \"kubernetes.io/projected/767860ff-f41a-4b9b-962c-f0083d0f069a-kube-api-access-82nzg\") pod \"node-exporter-69d5v\" (UID: \"767860ff-f41a-4b9b-962c-f0083d0f069a\") " pod="openshift-monitoring/node-exporter-69d5v" Jul 07 15:42:34.900923 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:34.900753 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/767860ff-f41a-4b9b-962c-f0083d0f069a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-69d5v\" (UID: \"767860ff-f41a-4b9b-962c-f0083d0f069a\") " pod="openshift-monitoring/node-exporter-69d5v" Jul 07 15:42:34.900923 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:34.900791 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/767860ff-f41a-4b9b-962c-f0083d0f069a-sys\") pod \"node-exporter-69d5v\" (UID: \"767860ff-f41a-4b9b-962c-f0083d0f069a\") " pod="openshift-monitoring/node-exporter-69d5v" Jul 07 15:42:34.900923 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:34.900910 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/767860ff-f41a-4b9b-962c-f0083d0f069a-node-exporter-textfile\") pod \"node-exporter-69d5v\" (UID: \"767860ff-f41a-4b9b-962c-f0083d0f069a\") " pod="openshift-monitoring/node-exporter-69d5v" Jul 07 15:42:34.901275 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:34.901254 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/767860ff-f41a-4b9b-962c-f0083d0f069a-metrics-client-ca\") pod \"node-exporter-69d5v\" (UID: \"767860ff-f41a-4b9b-962c-f0083d0f069a\") " pod="openshift-monitoring/node-exporter-69d5v" Jul 07 15:42:34.901324 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:34.901254 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/767860ff-f41a-4b9b-962c-f0083d0f069a-node-exporter-accelerators-collector-config\") pod \"node-exporter-69d5v\" (UID: \"767860ff-f41a-4b9b-962c-f0083d0f069a\") " pod="openshift-monitoring/node-exporter-69d5v" Jul 07 15:42:34.902793 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:34.902770 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/767860ff-f41a-4b9b-962c-f0083d0f069a-node-exporter-tls\") pod \"node-exporter-69d5v\" (UID: \"767860ff-f41a-4b9b-962c-f0083d0f069a\") " pod="openshift-monitoring/node-exporter-69d5v" Jul 07 15:42:34.902889 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:34.902876 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/767860ff-f41a-4b9b-962c-f0083d0f069a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-69d5v\" (UID: \"767860ff-f41a-4b9b-962c-f0083d0f069a\") " pod="openshift-monitoring/node-exporter-69d5v" Jul 07 15:42:34.910397 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:34.910376 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-82nzg\" (UniqueName: \"kubernetes.io/projected/767860ff-f41a-4b9b-962c-f0083d0f069a-kube-api-access-82nzg\") pod \"node-exporter-69d5v\" (UID: \"767860ff-f41a-4b9b-962c-f0083d0f069a\") " pod="openshift-monitoring/node-exporter-69d5v" Jul 07 15:42:34.919055 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:34.919034 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:42:34.984323 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:34.984243 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-69d5v" Jul 07 15:42:34.999170 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:42:34.999146 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod767860ff_f41a_4b9b_962c_f0083d0f069a.slice/crio-071f54e3e33e1d8e498176e2f47ff3b56804f2243cd1985abcaa34822a734474 WatchSource:0}: Error finding container 071f54e3e33e1d8e498176e2f47ff3b56804f2243cd1985abcaa34822a734474: Status 404 returned error can't find the container with id 071f54e3e33e1d8e498176e2f47ff3b56804f2243cd1985abcaa34822a734474 Jul 07 15:42:35.018663 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:35.018640 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:42:35.123751 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:35.123717 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:42:35.171081 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:35.171047 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-69d5v" event={"ID":"767860ff-f41a-4b9b-962c-f0083d0f069a","Type":"ContainerStarted","Data":"071f54e3e33e1d8e498176e2f47ff3b56804f2243cd1985abcaa34822a734474"} Jul 07 15:42:35.220797 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:35.220767 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:42:35.689508 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:35.689466 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:42:35.775400 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:35.775361 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:42:36.176654 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:36.176622 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-69d5v" event={"ID":"767860ff-f41a-4b9b-962c-f0083d0f069a","Type":"ContainerStarted","Data":"d2bbe224c4dab8a2eef35db404e4a46939646bf779964373aa70c9155d4bc867"} Jul 07 15:42:36.427917 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:36.427885 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:42:37.122688 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:37.122653 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:42:37.181415 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:37.181378 2565 generic.go:356] "Generic (PLEG): container finished" podID="767860ff-f41a-4b9b-962c-f0083d0f069a" containerID="d2bbe224c4dab8a2eef35db404e4a46939646bf779964373aa70c9155d4bc867" exitCode=0 Jul 07 15:42:37.181590 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:37.181461 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-69d5v" event={"ID":"767860ff-f41a-4b9b-962c-f0083d0f069a","Type":"ContainerDied","Data":"d2bbe224c4dab8a2eef35db404e4a46939646bf779964373aa70c9155d4bc867"} Jul 07 15:42:37.234545 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:37.234520 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:42:37.769580 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:37.769511 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:42:37.869710 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:37.869665 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:42:38.187617 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:38.187585 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-69d5v" event={"ID":"767860ff-f41a-4b9b-962c-f0083d0f069a","Type":"ContainerStarted","Data":"0385bb9cfdb09c4293ac8691b2d986e95b635f2c3cd368579115f240ffa64967"} Jul 07 15:42:38.188029 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:38.187665 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-69d5v" event={"ID":"767860ff-f41a-4b9b-962c-f0083d0f069a","Type":"ContainerStarted","Data":"86be73be1f88aece6479c640a511738b4ea2afdb8436eb1b0e20c30425ec1ad6"} Jul 07 15:42:38.189236 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:38.189213 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:42:38.213823 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:38.213776 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-69d5v" podStartSLOduration=3.091698862 podStartE2EDuration="4.213760182s" totalImagesPullingTime="1.055352197s" totalInitContainerRuntime="66.709123ms" isStatefulPod=true podCreationTimestamp="2026-07-07 15:42:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:42:38.211822998 +0000 UTC m=+97.168400466" watchObservedRunningTime="2026-07-07 15:42:38.213760182 +0000 UTC m=+97.170337649" Jul 07 15:42:38.298107 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:38.298049 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:42:38.914818 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:38.914785 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:42:39.017447 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:39.017418 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:42:39.564090 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:39.564050 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:42:39.672937 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:39.672901 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:42:39.772323 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:39.772278 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:42:39.868776 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:39.868680 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:42:40.968729 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:40.968697 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:42:41.065680 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:41.065653 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:42:41.193743 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:41.193698 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:42:41.381309 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:41.381273 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:42:41.461367 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:41.461312 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:42:42.119742 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:42.119713 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7f7d849b7d-bvxw4" Jul 07 15:42:42.333697 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:42.333653 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:42:42.748068 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:42.748040 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:42:44.078368 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:44.078339 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-94vlx" Jul 07 15:42:44.214667 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:44.214632 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:42:44.324711 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:44.324678 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:42:44.522626 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:44.522547 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:42:44.598613 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:44.598557 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:42:47.805749 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:47.805714 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:42:47.975768 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:47.975741 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:42:48.078416 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:48.078390 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:42:50.197704 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:50.197677 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:42:50.292123 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:50.292085 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:42:50.378763 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:50.378732 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:42:50.479388 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:50.479310 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:42:50.551130 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:50.551100 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:42:54.243403 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:54.243371 2565 generic.go:356] "Generic (PLEG): container finished" podID="17238627-b0d8-49ae-827e-cbf250ab8146" containerID="493b9985c3dccd03b09ee1f55cdcc36a9e73553ac48231ed8c57e2eeb272f9cc" exitCode=0 Jul 07 15:42:54.243782 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:54.243444 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5c847dcfc4-m4jd9" event={"ID":"17238627-b0d8-49ae-827e-cbf250ab8146","Type":"ContainerDied","Data":"493b9985c3dccd03b09ee1f55cdcc36a9e73553ac48231ed8c57e2eeb272f9cc"} Jul 07 15:42:54.243827 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:54.243797 2565 scope.go:117] "RemoveContainer" containerID="493b9985c3dccd03b09ee1f55cdcc36a9e73553ac48231ed8c57e2eeb272f9cc" Jul 07 15:42:55.248679 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:42:55.248643 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5c847dcfc4-m4jd9" event={"ID":"17238627-b0d8-49ae-827e-cbf250ab8146","Type":"ContainerStarted","Data":"150b3d875e896acff2b2d30ce56a02f0eae13887c8579161dfaed3463850140b"} Jul 07 15:43:09.683599 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:43:09.683559 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:43:09.981606 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:43:09.981530 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:43:10.495138 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:43:10.495105 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/301c1027-169c-47f5-a3ea-89cde78790d2-metrics-certs\") pod \"network-metrics-daemon-sdzxq\" (UID: \"301c1027-169c-47f5-a3ea-89cde78790d2\") " pod="openshift-multus/network-metrics-daemon-sdzxq" Jul 07 15:43:10.497402 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:43:10.497372 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/301c1027-169c-47f5-a3ea-89cde78790d2-metrics-certs\") pod \"network-metrics-daemon-sdzxq\" (UID: \"301c1027-169c-47f5-a3ea-89cde78790d2\") " pod="openshift-multus/network-metrics-daemon-sdzxq" Jul 07 15:43:10.774064 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:43:10.773981 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-t2q2r\"" Jul 07 15:43:10.781738 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:43:10.781711 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sdzxq" Jul 07 15:43:10.921144 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:43:10.920925 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-sdzxq"] Jul 07 15:43:10.923804 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:43:10.923776 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod301c1027_169c_47f5_a3ea_89cde78790d2.slice/crio-8de4280674de73aa341383e0b5fa5d73ea19bed65242fdd4b833dee8db6d48b0 WatchSource:0}: Error finding container 8de4280674de73aa341383e0b5fa5d73ea19bed65242fdd4b833dee8db6d48b0: Status 404 returned error can't find the container with id 8de4280674de73aa341383e0b5fa5d73ea19bed65242fdd4b833dee8db6d48b0 Jul 07 15:43:11.303769 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:43:11.303731 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-sdzxq" event={"ID":"301c1027-169c-47f5-a3ea-89cde78790d2","Type":"ContainerStarted","Data":"8de4280674de73aa341383e0b5fa5d73ea19bed65242fdd4b833dee8db6d48b0"} Jul 07 15:43:12.310005 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:43:12.309966 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-sdzxq" event={"ID":"301c1027-169c-47f5-a3ea-89cde78790d2","Type":"ContainerStarted","Data":"80628abc8b2ad4115dc44f9f1887b3733f07250d6e0bf6a8d0f99fd00f30b9e4"} Jul 07 15:43:12.310424 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:43:12.310064 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-sdzxq" event={"ID":"301c1027-169c-47f5-a3ea-89cde78790d2","Type":"ContainerStarted","Data":"bd354b5c0f754dde1d070dd0b54e71f0def8d3ffbf556ab3aa4cd3a039110010"} Jul 07 15:43:12.334423 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:43:12.334373 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-sdzxq" podStartSLOduration=130.417792011 podStartE2EDuration="2m11.334357274s" totalImagesPullingTime="916.565263ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:41:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:43:12.332829035 +0000 UTC m=+131.289406524" watchObservedRunningTime="2026-07-07 15:43:12.334357274 +0000 UTC m=+131.290934740" Jul 07 15:43:12.476123 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:43:12.476092 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:43:12.579546 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:43:12.579443 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:43:13.223374 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:43:13.223340 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:43:13.298743 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:43:13.298707 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:43:14.318131 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:43:14.318098 2565 generic.go:356] "Generic (PLEG): container finished" podID="394eb1a2-5a48-4813-b38b-227294ed15ba" containerID="d211acffe82cddb4ed78d961983111efbfdb31dc08460e53a2c9916c96ad0c12" exitCode=0 Jul 07 15:43:14.318526 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:43:14.318160 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-p6fgv" event={"ID":"394eb1a2-5a48-4813-b38b-227294ed15ba","Type":"ContainerDied","Data":"d211acffe82cddb4ed78d961983111efbfdb31dc08460e53a2c9916c96ad0c12"} Jul 07 15:43:14.318526 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:43:14.318442 2565 scope.go:117] "RemoveContainer" containerID="d211acffe82cddb4ed78d961983111efbfdb31dc08460e53a2c9916c96ad0c12" Jul 07 15:43:15.323079 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:43:15.323045 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-p6fgv" event={"ID":"394eb1a2-5a48-4813-b38b-227294ed15ba","Type":"ContainerStarted","Data":"76cb72d24d196303d0ab52eb6e03510857afa57fcd61112ab6f796e9352ddee0"} Jul 07 15:43:54.275372 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:43:54.275286 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:43:54.540394 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:43:54.540308 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:44:03.419655 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:44:03.419623 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:44:03.518530 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:44:03.518471 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:44:11.133040 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:44:11.133008 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:44:14.492545 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:44:14.492485 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:44:14.611561 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:44:14.611526 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:44:25.306854 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:44:25.306823 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:44:25.398639 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:44:25.398604 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:44:48.143309 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:44:48.143272 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:44:51.232356 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:44:51.232324 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:44:51.399428 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:44:51.399394 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:45:09.588360 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:45:09.588323 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:45:09.676764 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:45:09.676729 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:45:14.951291 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:45:14.951256 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:45:15.020620 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:45:15.020593 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:45:21.520474 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:45:21.520436 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:45:21.608562 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:45:21.608529 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:45:24.677609 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:45:24.677522 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:45:24.779128 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:45:24.779093 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:45:25.726988 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:45:25.726955 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:45:25.827940 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:45:25.827909 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:45:25.928299 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:45:25.928265 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:45:57.653320 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:45:57.653285 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:45:57.755066 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:45:57.755035 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:46:01.466190 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:46:01.466162 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-s9kl8_62da0165-d32c-4bba-9844-a49154d765e5/console-operator/1.log" Jul 07 15:46:01.471252 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:46:01.471223 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-942xh_31ff16f0-6054-4eba-942a-b07a5a7afac7/ovn-acl-logging/0.log" Jul 07 15:46:01.472072 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:46:01.472051 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-s9kl8_62da0165-d32c-4bba-9844-a49154d765e5/console-operator/1.log" Jul 07 15:46:01.477663 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:46:01.477642 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-942xh_31ff16f0-6054-4eba-942a-b07a5a7afac7/ovn-acl-logging/0.log" Jul 07 15:46:01.477799 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:46:01.477786 2565 kubelet.go:1648] "Image garbage collection succeeded" Jul 07 15:46:24.941704 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:46:24.941668 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:46:26.679993 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:46:26.679958 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:46:28.967201 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:46:28.967168 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:46:29.071449 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:46:29.071413 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:46:30.159551 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:46:30.159516 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:46:30.278776 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:46:30.278744 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:46:30.527643 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:46:30.527573 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:46:30.625739 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:46:30.625702 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:46:32.543354 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:46:32.543317 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:46:32.940217 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:46:32.940186 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:46:35.768445 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:46:35.768411 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:46:35.914420 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:46:35.914387 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:46:37.731070 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:46:37.731040 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:46:37.830045 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:46:37.830014 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:46:40.641331 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:46:40.641294 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:46:40.762729 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:46:40.762697 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:46:41.152231 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:46:41.152194 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:46:46.598483 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:46:46.598449 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:46:46.702657 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:46:46.702610 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:46:48.661555 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:46:48.661529 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:47:04.389924 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:04.389845 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:47:04.491538 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:04.491480 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:47:04.959587 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:04.959548 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:47:05.062886 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:05.062853 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:47:05.402447 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:05.402417 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:47:05.508785 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:05.508751 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:47:05.561336 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:05.561297 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-685dff6fcf-lmkgr"] Jul 07 15:47:05.564593 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:05.564570 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-685dff6fcf-lmkgr" Jul 07 15:47:05.568309 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:05.568290 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 07 15:47:05.568590 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:05.568574 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-kvnwl\"" Jul 07 15:47:05.569044 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:05.569028 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 07 15:47:05.569904 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:05.569888 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 07 15:47:05.571151 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:05.570963 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 07 15:47:05.594970 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:05.594935 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-685dff6fcf-lmkgr"] Jul 07 15:47:05.615462 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:05.615430 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhwrj\" (UniqueName: \"kubernetes.io/projected/fb691f38-96cb-4df1-bdaf-682a080bff74-kube-api-access-jhwrj\") pod \"opendatahub-operator-controller-manager-685dff6fcf-lmkgr\" (UID: \"fb691f38-96cb-4df1-bdaf-682a080bff74\") " pod="opendatahub/opendatahub-operator-controller-manager-685dff6fcf-lmkgr" Jul 07 15:47:05.615632 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:05.615519 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/fb691f38-96cb-4df1-bdaf-682a080bff74-apiservice-cert\") pod \"opendatahub-operator-controller-manager-685dff6fcf-lmkgr\" (UID: \"fb691f38-96cb-4df1-bdaf-682a080bff74\") " pod="opendatahub/opendatahub-operator-controller-manager-685dff6fcf-lmkgr" Jul 07 15:47:05.615632 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:05.615562 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/fb691f38-96cb-4df1-bdaf-682a080bff74-manifests\") pod \"opendatahub-operator-controller-manager-685dff6fcf-lmkgr\" (UID: \"fb691f38-96cb-4df1-bdaf-682a080bff74\") " pod="opendatahub/opendatahub-operator-controller-manager-685dff6fcf-lmkgr" Jul 07 15:47:05.615632 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:05.615608 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fb691f38-96cb-4df1-bdaf-682a080bff74-tmp\") pod \"opendatahub-operator-controller-manager-685dff6fcf-lmkgr\" (UID: \"fb691f38-96cb-4df1-bdaf-682a080bff74\") " pod="opendatahub/opendatahub-operator-controller-manager-685dff6fcf-lmkgr" Jul 07 15:47:05.615632 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:05.615623 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fb691f38-96cb-4df1-bdaf-682a080bff74-webhook-cert\") pod \"opendatahub-operator-controller-manager-685dff6fcf-lmkgr\" (UID: \"fb691f38-96cb-4df1-bdaf-682a080bff74\") " pod="opendatahub/opendatahub-operator-controller-manager-685dff6fcf-lmkgr" Jul 07 15:47:05.716875 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:05.716771 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhwrj\" (UniqueName: \"kubernetes.io/projected/fb691f38-96cb-4df1-bdaf-682a080bff74-kube-api-access-jhwrj\") pod \"opendatahub-operator-controller-manager-685dff6fcf-lmkgr\" (UID: \"fb691f38-96cb-4df1-bdaf-682a080bff74\") " pod="opendatahub/opendatahub-operator-controller-manager-685dff6fcf-lmkgr" Jul 07 15:47:05.716875 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:05.716826 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/fb691f38-96cb-4df1-bdaf-682a080bff74-apiservice-cert\") pod \"opendatahub-operator-controller-manager-685dff6fcf-lmkgr\" (UID: \"fb691f38-96cb-4df1-bdaf-682a080bff74\") " pod="opendatahub/opendatahub-operator-controller-manager-685dff6fcf-lmkgr" Jul 07 15:47:05.716875 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:05.716848 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/fb691f38-96cb-4df1-bdaf-682a080bff74-manifests\") pod \"opendatahub-operator-controller-manager-685dff6fcf-lmkgr\" (UID: \"fb691f38-96cb-4df1-bdaf-682a080bff74\") " pod="opendatahub/opendatahub-operator-controller-manager-685dff6fcf-lmkgr" Jul 07 15:47:05.717162 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:05.716880 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fb691f38-96cb-4df1-bdaf-682a080bff74-tmp\") pod \"opendatahub-operator-controller-manager-685dff6fcf-lmkgr\" (UID: \"fb691f38-96cb-4df1-bdaf-682a080bff74\") " pod="opendatahub/opendatahub-operator-controller-manager-685dff6fcf-lmkgr" Jul 07 15:47:05.717162 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:05.716898 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fb691f38-96cb-4df1-bdaf-682a080bff74-webhook-cert\") pod \"opendatahub-operator-controller-manager-685dff6fcf-lmkgr\" (UID: \"fb691f38-96cb-4df1-bdaf-682a080bff74\") " pod="opendatahub/opendatahub-operator-controller-manager-685dff6fcf-lmkgr" Jul 07 15:47:05.717368 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:05.717335 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/fb691f38-96cb-4df1-bdaf-682a080bff74-manifests\") pod \"opendatahub-operator-controller-manager-685dff6fcf-lmkgr\" (UID: \"fb691f38-96cb-4df1-bdaf-682a080bff74\") " pod="opendatahub/opendatahub-operator-controller-manager-685dff6fcf-lmkgr" Jul 07 15:47:05.719237 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:05.719213 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fb691f38-96cb-4df1-bdaf-682a080bff74-tmp\") pod \"opendatahub-operator-controller-manager-685dff6fcf-lmkgr\" (UID: \"fb691f38-96cb-4df1-bdaf-682a080bff74\") " pod="opendatahub/opendatahub-operator-controller-manager-685dff6fcf-lmkgr" Jul 07 15:47:05.719335 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:05.719294 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fb691f38-96cb-4df1-bdaf-682a080bff74-webhook-cert\") pod \"opendatahub-operator-controller-manager-685dff6fcf-lmkgr\" (UID: \"fb691f38-96cb-4df1-bdaf-682a080bff74\") " pod="opendatahub/opendatahub-operator-controller-manager-685dff6fcf-lmkgr" Jul 07 15:47:05.719385 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:05.719354 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/fb691f38-96cb-4df1-bdaf-682a080bff74-apiservice-cert\") pod \"opendatahub-operator-controller-manager-685dff6fcf-lmkgr\" (UID: \"fb691f38-96cb-4df1-bdaf-682a080bff74\") " pod="opendatahub/opendatahub-operator-controller-manager-685dff6fcf-lmkgr" Jul 07 15:47:05.732640 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:05.732613 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhwrj\" (UniqueName: \"kubernetes.io/projected/fb691f38-96cb-4df1-bdaf-682a080bff74-kube-api-access-jhwrj\") pod \"opendatahub-operator-controller-manager-685dff6fcf-lmkgr\" (UID: \"fb691f38-96cb-4df1-bdaf-682a080bff74\") " pod="opendatahub/opendatahub-operator-controller-manager-685dff6fcf-lmkgr" Jul 07 15:47:05.798119 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:05.798086 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:47:05.874780 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:05.874742 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-685dff6fcf-lmkgr" Jul 07 15:47:05.892341 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:05.892308 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:47:05.998261 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:05.998190 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:47:06.018907 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:06.018875 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-685dff6fcf-lmkgr"] Jul 07 15:47:06.022751 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:47:06.022724 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb691f38_96cb_4df1_bdaf_682a080bff74.slice/crio-99bacc8b0bba82fb58efa00aad820166d0074bba816d4644d3fa8572899ae961 WatchSource:0}: Error finding container 99bacc8b0bba82fb58efa00aad820166d0074bba816d4644d3fa8572899ae961: Status 404 returned error can't find the container with id 99bacc8b0bba82fb58efa00aad820166d0074bba816d4644d3fa8572899ae961 Jul 07 15:47:06.024370 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:06.024353 2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 07 15:47:06.112971 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:06.112938 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-685dff6fcf-lmkgr" event={"ID":"fb691f38-96cb-4df1-bdaf-682a080bff74","Type":"ContainerStarted","Data":"99bacc8b0bba82fb58efa00aad820166d0074bba816d4644d3fa8572899ae961"} Jul 07 15:47:06.131998 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:06.131969 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:47:06.236541 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:06.236486 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:47:06.337650 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:06.337618 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:47:09.125597 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:09.125555 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-685dff6fcf-lmkgr" event={"ID":"fb691f38-96cb-4df1-bdaf-682a080bff74","Type":"ContainerStarted","Data":"7aa7af417a75afeb6ea50d97a51d4b208fbeedb51592d590dfc037012c5579e9"} Jul 07 15:47:10.130702 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:10.130611 2565 generic.go:356] "Generic (PLEG): container finished" podID="fb691f38-96cb-4df1-bdaf-682a080bff74" containerID="7aa7af417a75afeb6ea50d97a51d4b208fbeedb51592d590dfc037012c5579e9" exitCode=0 Jul 07 15:47:10.131101 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:10.130694 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-685dff6fcf-lmkgr" event={"ID":"fb691f38-96cb-4df1-bdaf-682a080bff74","Type":"ContainerDied","Data":"7aa7af417a75afeb6ea50d97a51d4b208fbeedb51592d590dfc037012c5579e9"} Jul 07 15:47:11.136578 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:11.136544 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-685dff6fcf-lmkgr" event={"ID":"fb691f38-96cb-4df1-bdaf-682a080bff74","Type":"ContainerStarted","Data":"4601915f9e96c4b1098258d84db1bfe351ed3702fc0fc77c73d5629f81671b63"} Jul 07 15:47:11.136970 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:11.136703 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-685dff6fcf-lmkgr" Jul 07 15:47:11.170210 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:11.170149 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-685dff6fcf-lmkgr" podStartSLOduration=1.982342657 podStartE2EDuration="6.170127828s" totalImagesPullingTime="3.16638968s" totalInitContainerRuntime="1.021395491s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:47:05 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:47:11.168445736 +0000 UTC m=+370.125023205" watchObservedRunningTime="2026-07-07 15:47:11.170127828 +0000 UTC m=+370.126705297" Jul 07 15:47:22.142836 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:22.142809 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-685dff6fcf-lmkgr" Jul 07 15:47:25.825191 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:25.825153 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:47:26.179053 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:26.178951 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:47:26.620734 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:26.620701 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:47:26.788281 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:26.788253 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-ccc59c597-jlq44"] Jul 07 15:47:26.790356 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:26.790338 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-ccc59c597-jlq44" Jul 07 15:47:26.799052 ip-10-0-132-65 kubenswrapper[2565]: E0707 15:47:26.799018 2565 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"kube-auth-proxy-creds\" is forbidden: User \"system:node:ip-10-0-132-65.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-132-65.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" type="*v1.Secret" Jul 07 15:47:26.800551 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:26.800531 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-6tbg6\"" Jul 07 15:47:26.817581 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:26.817552 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 07 15:47:26.829614 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:26.829586 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-ccc59c597-jlq44"] Jul 07 15:47:26.887546 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:26.887449 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/21bfa646-dd2e-40c2-847c-139c13e56c53-tmp\") pod \"kube-auth-proxy-ccc59c597-jlq44\" (UID: \"21bfa646-dd2e-40c2-847c-139c13e56c53\") " pod="openshift-ingress/kube-auth-proxy-ccc59c597-jlq44" Jul 07 15:47:26.887546 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:26.887511 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21bfa646-dd2e-40c2-847c-139c13e56c53-tls-certs\") pod \"kube-auth-proxy-ccc59c597-jlq44\" (UID: \"21bfa646-dd2e-40c2-847c-139c13e56c53\") " pod="openshift-ingress/kube-auth-proxy-ccc59c597-jlq44" Jul 07 15:47:26.887750 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:26.887644 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjddr\" (UniqueName: \"kubernetes.io/projected/21bfa646-dd2e-40c2-847c-139c13e56c53-kube-api-access-cjddr\") pod \"kube-auth-proxy-ccc59c597-jlq44\" (UID: \"21bfa646-dd2e-40c2-847c-139c13e56c53\") " pod="openshift-ingress/kube-auth-proxy-ccc59c597-jlq44" Jul 07 15:47:26.905990 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:26.905952 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-4wkpn"] Jul 07 15:47:26.908411 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:26.908392 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-4wkpn" Jul 07 15:47:26.911522 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:26.911505 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Jul 07 15:47:26.912421 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:26.912404 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-86kqb\"" Jul 07 15:47:26.952254 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:26.952229 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-4wkpn"] Jul 07 15:47:26.988262 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:26.988228 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjddr\" (UniqueName: \"kubernetes.io/projected/21bfa646-dd2e-40c2-847c-139c13e56c53-kube-api-access-cjddr\") pod \"kube-auth-proxy-ccc59c597-jlq44\" (UID: \"21bfa646-dd2e-40c2-847c-139c13e56c53\") " pod="openshift-ingress/kube-auth-proxy-ccc59c597-jlq44" Jul 07 15:47:26.988458 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:26.988279 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/21bfa646-dd2e-40c2-847c-139c13e56c53-tmp\") pod \"kube-auth-proxy-ccc59c597-jlq44\" (UID: \"21bfa646-dd2e-40c2-847c-139c13e56c53\") " pod="openshift-ingress/kube-auth-proxy-ccc59c597-jlq44" Jul 07 15:47:26.988458 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:26.988339 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/96ac3c97-7a14-441e-a092-3cd28b8c2b13-cert\") pod \"odh-model-controller-7998c67dc4-4wkpn\" (UID: \"96ac3c97-7a14-441e-a092-3cd28b8c2b13\") " pod="opendatahub/odh-model-controller-7998c67dc4-4wkpn" Jul 07 15:47:26.988458 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:26.988370 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xf86\" (UniqueName: \"kubernetes.io/projected/96ac3c97-7a14-441e-a092-3cd28b8c2b13-kube-api-access-8xf86\") pod \"odh-model-controller-7998c67dc4-4wkpn\" (UID: \"96ac3c97-7a14-441e-a092-3cd28b8c2b13\") " pod="opendatahub/odh-model-controller-7998c67dc4-4wkpn" Jul 07 15:47:26.988458 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:26.988408 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21bfa646-dd2e-40c2-847c-139c13e56c53-tls-certs\") pod \"kube-auth-proxy-ccc59c597-jlq44\" (UID: \"21bfa646-dd2e-40c2-847c-139c13e56c53\") " pod="openshift-ingress/kube-auth-proxy-ccc59c597-jlq44" Jul 07 15:47:26.991064 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:26.991041 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21bfa646-dd2e-40c2-847c-139c13e56c53-tls-certs\") pod \"kube-auth-proxy-ccc59c597-jlq44\" (UID: \"21bfa646-dd2e-40c2-847c-139c13e56c53\") " pod="openshift-ingress/kube-auth-proxy-ccc59c597-jlq44" Jul 07 15:47:26.991181 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:26.991161 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/21bfa646-dd2e-40c2-847c-139c13e56c53-tmp\") pod \"kube-auth-proxy-ccc59c597-jlq44\" (UID: \"21bfa646-dd2e-40c2-847c-139c13e56c53\") " pod="openshift-ingress/kube-auth-proxy-ccc59c597-jlq44" Jul 07 15:47:27.018030 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:27.017998 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:47:27.018274 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:27.018251 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjddr\" (UniqueName: \"kubernetes.io/projected/21bfa646-dd2e-40c2-847c-139c13e56c53-kube-api-access-cjddr\") pod \"kube-auth-proxy-ccc59c597-jlq44\" (UID: \"21bfa646-dd2e-40c2-847c-139c13e56c53\") " pod="openshift-ingress/kube-auth-proxy-ccc59c597-jlq44" Jul 07 15:47:27.089827 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:27.089782 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/96ac3c97-7a14-441e-a092-3cd28b8c2b13-cert\") pod \"odh-model-controller-7998c67dc4-4wkpn\" (UID: \"96ac3c97-7a14-441e-a092-3cd28b8c2b13\") " pod="opendatahub/odh-model-controller-7998c67dc4-4wkpn" Jul 07 15:47:27.089827 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:27.089822 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xf86\" (UniqueName: \"kubernetes.io/projected/96ac3c97-7a14-441e-a092-3cd28b8c2b13-kube-api-access-8xf86\") pod \"odh-model-controller-7998c67dc4-4wkpn\" (UID: \"96ac3c97-7a14-441e-a092-3cd28b8c2b13\") " pod="opendatahub/odh-model-controller-7998c67dc4-4wkpn" Jul 07 15:47:27.090088 ip-10-0-132-65 kubenswrapper[2565]: E0707 15:47:27.089932 2565 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found Jul 07 15:47:27.090088 ip-10-0-132-65 kubenswrapper[2565]: E0707 15:47:27.090003 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/96ac3c97-7a14-441e-a092-3cd28b8c2b13-cert podName:96ac3c97-7a14-441e-a092-3cd28b8c2b13 nodeName:}" failed. No retries permitted until 2026-07-07 15:47:27.589988097 +0000 UTC m=+386.546565546 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/96ac3c97-7a14-441e-a092-3cd28b8c2b13-cert") pod "odh-model-controller-7998c67dc4-4wkpn" (UID: "96ac3c97-7a14-441e-a092-3cd28b8c2b13") : secret "odh-model-controller-webhook-cert" not found Jul 07 15:47:27.099778 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:27.099743 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-ccc59c597-jlq44" Jul 07 15:47:27.123129 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:27.123103 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xf86\" (UniqueName: \"kubernetes.io/projected/96ac3c97-7a14-441e-a092-3cd28b8c2b13-kube-api-access-8xf86\") pod \"odh-model-controller-7998c67dc4-4wkpn\" (UID: \"96ac3c97-7a14-441e-a092-3cd28b8c2b13\") " pod="opendatahub/odh-model-controller-7998c67dc4-4wkpn" Jul 07 15:47:27.206380 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:27.206335 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:47:27.336367 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:27.336335 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-ccc59c597-jlq44"] Jul 07 15:47:27.340438 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:47:27.340407 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21bfa646_dd2e_40c2_847c_139c13e56c53.slice/crio-b2ed5b09390a80df65307eb70cd925f70d419660e1b07947d8467f5585448803 WatchSource:0}: Error finding container b2ed5b09390a80df65307eb70cd925f70d419660e1b07947d8467f5585448803: Status 404 returned error can't find the container with id b2ed5b09390a80df65307eb70cd925f70d419660e1b07947d8467f5585448803 Jul 07 15:47:27.594399 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:27.594365 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/96ac3c97-7a14-441e-a092-3cd28b8c2b13-cert\") pod \"odh-model-controller-7998c67dc4-4wkpn\" (UID: \"96ac3c97-7a14-441e-a092-3cd28b8c2b13\") " pod="opendatahub/odh-model-controller-7998c67dc4-4wkpn" Jul 07 15:47:27.596815 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:27.596788 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/96ac3c97-7a14-441e-a092-3cd28b8c2b13-cert\") pod \"odh-model-controller-7998c67dc4-4wkpn\" (UID: \"96ac3c97-7a14-441e-a092-3cd28b8c2b13\") " pod="opendatahub/odh-model-controller-7998c67dc4-4wkpn" Jul 07 15:47:27.646804 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:27.646771 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:47:27.818689 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:27.818655 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-4wkpn" Jul 07 15:47:27.906468 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:27.906414 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:47:28.007620 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:28.007593 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-4wkpn"] Jul 07 15:47:28.008877 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:47:28.008844 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96ac3c97_7a14_441e_a092_3cd28b8c2b13.slice/crio-ed75a85e6c4be293430b78cacaf7824f03328aa69096b9e3db5ddba034c8b8a1 WatchSource:0}: Error finding container ed75a85e6c4be293430b78cacaf7824f03328aa69096b9e3db5ddba034c8b8a1: Status 404 returned error can't find the container with id ed75a85e6c4be293430b78cacaf7824f03328aa69096b9e3db5ddba034c8b8a1 Jul 07 15:47:28.203271 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:28.203173 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-4wkpn" event={"ID":"96ac3c97-7a14-441e-a092-3cd28b8c2b13","Type":"ContainerStarted","Data":"ed75a85e6c4be293430b78cacaf7824f03328aa69096b9e3db5ddba034c8b8a1"} Jul 07 15:47:28.204770 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:28.204731 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-ccc59c597-jlq44" event={"ID":"21bfa646-dd2e-40c2-847c-139c13e56c53","Type":"ContainerStarted","Data":"b2ed5b09390a80df65307eb70cd925f70d419660e1b07947d8467f5585448803"} Jul 07 15:47:28.288579 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:28.288540 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 07 15:47:28.348832 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:28.348803 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:47:28.498562 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:28.498469 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:47:29.021716 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:29.021628 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:47:29.107903 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:29.107871 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-q85t9"] Jul 07 15:47:29.111004 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:29.110535 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-q85t9" Jul 07 15:47:29.127806 ip-10-0-132-65 kubenswrapper[2565]: E0707 15:47:29.127749 2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"model-serving-api-54d89bd5fc-q85t9\" is forbidden: User \"system:node:ip-10-0-132-65.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-65.ec2.internal' and this object" podUID="338a6794-f166-47b8-929e-14332af6e1bf" pod="opendatahub/model-serving-api-54d89bd5fc-q85t9" Jul 07 15:47:29.127806 ip-10-0-132-65 kubenswrapper[2565]: E0707 15:47:29.127763 2565 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"model-serving-api-tls\" is forbidden: User \"system:node:ip-10-0-132-65.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-65.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" type="*v1.Secret" Jul 07 15:47:29.127806 ip-10-0-132-65 kubenswrapper[2565]: E0707 15:47:29.127796 2565 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"model-serving-api-dockercfg-hznpw\" is forbidden: User \"system:node:ip-10-0-132-65.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-65.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-hznpw\"" type="*v1.Secret" Jul 07 15:47:29.210256 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:29.210086 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/338a6794-f166-47b8-929e-14332af6e1bf-tls-certs\") pod \"model-serving-api-54d89bd5fc-q85t9\" (UID: \"338a6794-f166-47b8-929e-14332af6e1bf\") " pod="opendatahub/model-serving-api-54d89bd5fc-q85t9" Jul 07 15:47:29.210256 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:29.210178 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6czpj\" (UniqueName: \"kubernetes.io/projected/338a6794-f166-47b8-929e-14332af6e1bf-kube-api-access-6czpj\") pod \"model-serving-api-54d89bd5fc-q85t9\" (UID: \"338a6794-f166-47b8-929e-14332af6e1bf\") " pod="opendatahub/model-serving-api-54d89bd5fc-q85t9" Jul 07 15:47:29.228813 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:29.228760 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-q85t9"] Jul 07 15:47:29.311345 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:29.310901 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/338a6794-f166-47b8-929e-14332af6e1bf-tls-certs\") pod \"model-serving-api-54d89bd5fc-q85t9\" (UID: \"338a6794-f166-47b8-929e-14332af6e1bf\") " pod="opendatahub/model-serving-api-54d89bd5fc-q85t9" Jul 07 15:47:29.311345 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:29.310989 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6czpj\" (UniqueName: \"kubernetes.io/projected/338a6794-f166-47b8-929e-14332af6e1bf-kube-api-access-6czpj\") pod \"model-serving-api-54d89bd5fc-q85t9\" (UID: \"338a6794-f166-47b8-929e-14332af6e1bf\") " pod="opendatahub/model-serving-api-54d89bd5fc-q85t9" Jul 07 15:47:29.328428 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:29.328387 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6czpj\" (UniqueName: \"kubernetes.io/projected/338a6794-f166-47b8-929e-14332af6e1bf-kube-api-access-6czpj\") pod \"model-serving-api-54d89bd5fc-q85t9\" (UID: \"338a6794-f166-47b8-929e-14332af6e1bf\") " pod="opendatahub/model-serving-api-54d89bd5fc-q85t9" Jul 07 15:47:29.674700 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:29.674612 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:47:29.779547 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:29.779512 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:47:29.792204 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:29.792174 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-xmjjf"] Jul 07 15:47:29.800131 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:29.800102 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-xmjjf" Jul 07 15:47:29.803611 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:29.803238 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 07 15:47:29.803611 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:29.803363 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-gq5pn\"" Jul 07 15:47:29.803611 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:29.803517 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 07 15:47:29.803611 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:29.803606 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 07 15:47:29.804023 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:29.803999 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 07 15:47:29.820124 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:29.820098 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-xmjjf"] Jul 07 15:47:29.893772 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:29.893739 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:47:29.916893 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:29.916834 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/44256b42-3d30-4312-be20-ae4db826e66c-local-certs\") pod \"istiod-openshift-gateway-7574459656-xmjjf\" (UID: \"44256b42-3d30-4312-be20-ae4db826e66c\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-xmjjf" Jul 07 15:47:29.916893 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:29.916892 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/44256b42-3d30-4312-be20-ae4db826e66c-cacerts\") pod \"istiod-openshift-gateway-7574459656-xmjjf\" (UID: \"44256b42-3d30-4312-be20-ae4db826e66c\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-xmjjf" Jul 07 15:47:29.917127 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:29.916924 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44256b42-3d30-4312-be20-ae4db826e66c-istio-token\") pod \"istiod-openshift-gateway-7574459656-xmjjf\" (UID: \"44256b42-3d30-4312-be20-ae4db826e66c\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-xmjjf" Jul 07 15:47:29.917127 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:29.916948 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sb2xq\" (UniqueName: \"kubernetes.io/projected/44256b42-3d30-4312-be20-ae4db826e66c-kube-api-access-sb2xq\") pod \"istiod-openshift-gateway-7574459656-xmjjf\" (UID: \"44256b42-3d30-4312-be20-ae4db826e66c\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-xmjjf" Jul 07 15:47:29.917127 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:29.917040 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/44256b42-3d30-4312-be20-ae4db826e66c-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-xmjjf\" (UID: \"44256b42-3d30-4312-be20-ae4db826e66c\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-xmjjf" Jul 07 15:47:29.917127 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:29.917086 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/44256b42-3d30-4312-be20-ae4db826e66c-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-xmjjf\" (UID: \"44256b42-3d30-4312-be20-ae4db826e66c\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-xmjjf" Jul 07 15:47:29.917351 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:29.917190 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/44256b42-3d30-4312-be20-ae4db826e66c-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-xmjjf\" (UID: \"44256b42-3d30-4312-be20-ae4db826e66c\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-xmjjf" Jul 07 15:47:30.017741 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:30.017664 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/44256b42-3d30-4312-be20-ae4db826e66c-local-certs\") pod \"istiod-openshift-gateway-7574459656-xmjjf\" (UID: \"44256b42-3d30-4312-be20-ae4db826e66c\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-xmjjf" Jul 07 15:47:30.017741 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:30.017709 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/44256b42-3d30-4312-be20-ae4db826e66c-cacerts\") pod \"istiod-openshift-gateway-7574459656-xmjjf\" (UID: \"44256b42-3d30-4312-be20-ae4db826e66c\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-xmjjf" Jul 07 15:47:30.017741 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:30.017729 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44256b42-3d30-4312-be20-ae4db826e66c-istio-token\") pod \"istiod-openshift-gateway-7574459656-xmjjf\" (UID: \"44256b42-3d30-4312-be20-ae4db826e66c\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-xmjjf" Jul 07 15:47:30.018005 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:30.017756 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sb2xq\" (UniqueName: \"kubernetes.io/projected/44256b42-3d30-4312-be20-ae4db826e66c-kube-api-access-sb2xq\") pod \"istiod-openshift-gateway-7574459656-xmjjf\" (UID: \"44256b42-3d30-4312-be20-ae4db826e66c\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-xmjjf" Jul 07 15:47:30.018005 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:30.017794 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/44256b42-3d30-4312-be20-ae4db826e66c-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-xmjjf\" (UID: \"44256b42-3d30-4312-be20-ae4db826e66c\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-xmjjf" Jul 07 15:47:30.018005 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:30.017823 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/44256b42-3d30-4312-be20-ae4db826e66c-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-xmjjf\" (UID: \"44256b42-3d30-4312-be20-ae4db826e66c\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-xmjjf" Jul 07 15:47:30.018005 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:30.017875 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/44256b42-3d30-4312-be20-ae4db826e66c-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-xmjjf\" (UID: \"44256b42-3d30-4312-be20-ae4db826e66c\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-xmjjf" Jul 07 15:47:30.018695 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:30.018655 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/44256b42-3d30-4312-be20-ae4db826e66c-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-xmjjf\" (UID: \"44256b42-3d30-4312-be20-ae4db826e66c\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-xmjjf" Jul 07 15:47:30.020441 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:30.020414 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/44256b42-3d30-4312-be20-ae4db826e66c-local-certs\") pod \"istiod-openshift-gateway-7574459656-xmjjf\" (UID: \"44256b42-3d30-4312-be20-ae4db826e66c\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-xmjjf" Jul 07 15:47:30.020810 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:30.020786 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/44256b42-3d30-4312-be20-ae4db826e66c-cacerts\") pod \"istiod-openshift-gateway-7574459656-xmjjf\" (UID: \"44256b42-3d30-4312-be20-ae4db826e66c\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-xmjjf" Jul 07 15:47:30.020810 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:30.020803 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/44256b42-3d30-4312-be20-ae4db826e66c-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-xmjjf\" (UID: \"44256b42-3d30-4312-be20-ae4db826e66c\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-xmjjf" Jul 07 15:47:30.020970 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:30.020911 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/44256b42-3d30-4312-be20-ae4db826e66c-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-xmjjf\" (UID: \"44256b42-3d30-4312-be20-ae4db826e66c\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-xmjjf" Jul 07 15:47:30.043987 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:30.043962 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44256b42-3d30-4312-be20-ae4db826e66c-istio-token\") pod \"istiod-openshift-gateway-7574459656-xmjjf\" (UID: \"44256b42-3d30-4312-be20-ae4db826e66c\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-xmjjf" Jul 07 15:47:30.044307 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:30.044249 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sb2xq\" (UniqueName: \"kubernetes.io/projected/44256b42-3d30-4312-be20-ae4db826e66c-kube-api-access-sb2xq\") pod \"istiod-openshift-gateway-7574459656-xmjjf\" (UID: \"44256b42-3d30-4312-be20-ae4db826e66c\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-xmjjf" Jul 07 15:47:30.102652 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:30.102616 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Jul 07 15:47:30.112512 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:30.112463 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-xmjjf" Jul 07 15:47:30.114888 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:30.114861 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/338a6794-f166-47b8-929e-14332af6e1bf-tls-certs\") pod \"model-serving-api-54d89bd5fc-q85t9\" (UID: \"338a6794-f166-47b8-929e-14332af6e1bf\") " pod="opendatahub/model-serving-api-54d89bd5fc-q85t9" Jul 07 15:47:30.184089 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:30.184052 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-hznpw\"" Jul 07 15:47:30.324036 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:30.324006 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-q85t9" Jul 07 15:47:31.090288 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:31.090252 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-q85t9"] Jul 07 15:47:31.092110 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:47:31.092075 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod338a6794_f166_47b8_929e_14332af6e1bf.slice/crio-e9ec624a5389b03df26319dc722241252d7aa7d8a5a7385fa1f2a9c83e363f39 WatchSource:0}: Error finding container e9ec624a5389b03df26319dc722241252d7aa7d8a5a7385fa1f2a9c83e363f39: Status 404 returned error can't find the container with id e9ec624a5389b03df26319dc722241252d7aa7d8a5a7385fa1f2a9c83e363f39 Jul 07 15:47:31.118712 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:31.118686 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-xmjjf"] Jul 07 15:47:31.120926 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:47:31.120886 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44256b42_3d30_4312_be20_ae4db826e66c.slice/crio-ecfdc1da3a14523478b0f87484f16ee04096518def3e82e1c282c317f92d1708 WatchSource:0}: Error finding container ecfdc1da3a14523478b0f87484f16ee04096518def3e82e1c282c317f92d1708: Status 404 returned error can't find the container with id ecfdc1da3a14523478b0f87484f16ee04096518def3e82e1c282c317f92d1708 Jul 07 15:47:31.219755 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:31.219653 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-xmjjf" event={"ID":"44256b42-3d30-4312-be20-ae4db826e66c","Type":"ContainerStarted","Data":"ecfdc1da3a14523478b0f87484f16ee04096518def3e82e1c282c317f92d1708"} Jul 07 15:47:31.221096 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:31.221070 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-4wkpn" event={"ID":"96ac3c97-7a14-441e-a092-3cd28b8c2b13","Type":"ContainerStarted","Data":"35b673c8ebe74a4c29b18927ec848e413075ee425f7bc426f4132cb8199144b4"} Jul 07 15:47:31.221220 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:31.221182 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-4wkpn" Jul 07 15:47:31.222655 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:31.222632 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-ccc59c597-jlq44" event={"ID":"21bfa646-dd2e-40c2-847c-139c13e56c53","Type":"ContainerStarted","Data":"88e8c47541193ac3804110d917fe6848112385429f32ec06a312c7543b878708"} Jul 07 15:47:31.223706 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:31.223687 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-q85t9" event={"ID":"338a6794-f166-47b8-929e-14332af6e1bf","Type":"ContainerStarted","Data":"e9ec624a5389b03df26319dc722241252d7aa7d8a5a7385fa1f2a9c83e363f39"} Jul 07 15:47:31.246264 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:31.246213 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-4wkpn" podStartSLOduration=2.310915938 podStartE2EDuration="5.246197482s" totalImagesPullingTime="2.935281544s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:47:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:47:31.245704869 +0000 UTC m=+390.202282327" watchObservedRunningTime="2026-07-07 15:47:31.246197482 +0000 UTC m=+390.202774951" Jul 07 15:47:31.279689 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:31.279640 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-ccc59c597-jlq44" podStartSLOduration=1.724491963 podStartE2EDuration="5.279620342s" totalImagesPullingTime="3.555128379s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:47:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:47:31.275007883 +0000 UTC m=+390.231585351" watchObservedRunningTime="2026-07-07 15:47:31.279620342 +0000 UTC m=+390.236197808" Jul 07 15:47:31.824658 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:31.824619 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:47:33.233838 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:33.233803 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-q85t9" event={"ID":"338a6794-f166-47b8-929e-14332af6e1bf","Type":"ContainerStarted","Data":"74848e95a787cf8efeedae143de5c2f63aaf8ae03f0badb9da383c7c26cc6391"} Jul 07 15:47:33.234198 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:33.233956 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-q85t9" Jul 07 15:47:33.254000 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:33.253943 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-q85t9" podStartSLOduration=2.571393383 podStartE2EDuration="4.253926106s" totalImagesPullingTime="1.682532723s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:47:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:47:33.252535315 +0000 UTC m=+392.209112782" watchObservedRunningTime="2026-07-07 15:47:33.253926106 +0000 UTC m=+392.210503572" Jul 07 15:47:37.378626 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:37.378572 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 07 15:47:37.378887 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:37.378675 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 07 15:47:38.260756 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:38.260704 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-xmjjf" event={"ID":"44256b42-3d30-4312-be20-ae4db826e66c","Type":"ContainerStarted","Data":"05397023ac0329ff9a038bb23cfaf599b2e4258d14551e820aa939dff2f4e381"} Jul 07 15:47:38.261287 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:38.261256 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-xmjjf" Jul 07 15:47:38.262473 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:38.262433 2565 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7574459656-xmjjf container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 07 15:47:38.262608 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:38.262518 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7574459656-xmjjf" podUID="44256b42-3d30-4312-be20-ae4db826e66c" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 07 15:47:38.305289 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:38.305226 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7574459656-xmjjf" podStartSLOduration=3.049945919 podStartE2EDuration="9.305204403s" totalImagesPullingTime="6.255258484s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:47:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:47:38.3050786 +0000 UTC m=+397.261656069" watchObservedRunningTime="2026-07-07 15:47:38.305204403 +0000 UTC m=+397.261781869" Jul 07 15:47:38.913674 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:38.913636 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:47:39.050612 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:39.050577 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:47:39.164086 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:39.164000 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:47:39.267304 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:39.267267 2565 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7574459656-xmjjf container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 07 15:47:39.267520 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:39.267328 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7574459656-xmjjf" podUID="44256b42-3d30-4312-be20-ae4db826e66c" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 07 15:47:39.330825 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:39.330793 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:47:39.437296 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:39.437224 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-l6mcc"] Jul 07 15:47:39.440347 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:39.440329 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-l6mcc" Jul 07 15:47:39.444539 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:39.444472 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-gz46p"] Jul 07 15:47:39.447014 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:39.446995 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Jul 07 15:47:39.447273 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:39.447228 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-gz46p" Jul 07 15:47:39.447349 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:39.447279 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-xdlk8\"" Jul 07 15:47:39.452986 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:39.452965 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:47:39.459609 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:39.459590 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-fgshn\"" Jul 07 15:47:39.460053 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:39.460038 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Jul 07 15:47:39.473178 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:39.473148 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-l6mcc"] Jul 07 15:47:39.487407 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:39.487381 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-gz46p"] Jul 07 15:47:39.504185 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:39.504159 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/adac090f-b466-4ea0-b254-ec2d65dd63af-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-l6mcc\" (UID: \"adac090f-b466-4ea0-b254-ec2d65dd63af\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-l6mcc" Jul 07 15:47:39.504303 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:39.504193 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8a5048fd-5651-491f-9cea-7f4d93706202-cert\") pod \"kserve-controller-manager-d6f4b65b8-gz46p\" (UID: \"8a5048fd-5651-491f-9cea-7f4d93706202\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-gz46p" Jul 07 15:47:39.504303 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:39.504236 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phc89\" (UniqueName: \"kubernetes.io/projected/8a5048fd-5651-491f-9cea-7f4d93706202-kube-api-access-phc89\") pod \"kserve-controller-manager-d6f4b65b8-gz46p\" (UID: \"8a5048fd-5651-491f-9cea-7f4d93706202\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-gz46p" Jul 07 15:47:39.504303 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:39.504298 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gfx8b\" (UniqueName: \"kubernetes.io/projected/adac090f-b466-4ea0-b254-ec2d65dd63af-kube-api-access-gfx8b\") pod \"llmisvc-controller-manager-777c6bd7d5-l6mcc\" (UID: \"adac090f-b466-4ea0-b254-ec2d65dd63af\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-l6mcc" Jul 07 15:47:39.605624 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:39.605584 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfx8b\" (UniqueName: \"kubernetes.io/projected/adac090f-b466-4ea0-b254-ec2d65dd63af-kube-api-access-gfx8b\") pod \"llmisvc-controller-manager-777c6bd7d5-l6mcc\" (UID: \"adac090f-b466-4ea0-b254-ec2d65dd63af\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-l6mcc" Jul 07 15:47:39.605624 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:39.605632 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/adac090f-b466-4ea0-b254-ec2d65dd63af-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-l6mcc\" (UID: \"adac090f-b466-4ea0-b254-ec2d65dd63af\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-l6mcc" Jul 07 15:47:39.605982 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:39.605659 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8a5048fd-5651-491f-9cea-7f4d93706202-cert\") pod \"kserve-controller-manager-d6f4b65b8-gz46p\" (UID: \"8a5048fd-5651-491f-9cea-7f4d93706202\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-gz46p" Jul 07 15:47:39.605982 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:39.605743 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phc89\" (UniqueName: \"kubernetes.io/projected/8a5048fd-5651-491f-9cea-7f4d93706202-kube-api-access-phc89\") pod \"kserve-controller-manager-d6f4b65b8-gz46p\" (UID: \"8a5048fd-5651-491f-9cea-7f4d93706202\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-gz46p" Jul 07 15:47:39.605982 ip-10-0-132-65 kubenswrapper[2565]: E0707 15:47:39.605840 2565 secret.go:189] Couldn't get secret opendatahub/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found Jul 07 15:47:39.605982 ip-10-0-132-65 kubenswrapper[2565]: E0707 15:47:39.605906 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/adac090f-b466-4ea0-b254-ec2d65dd63af-cert podName:adac090f-b466-4ea0-b254-ec2d65dd63af nodeName:}" failed. No retries permitted until 2026-07-07 15:47:40.10588614 +0000 UTC m=+399.062463598 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/adac090f-b466-4ea0-b254-ec2d65dd63af-cert") pod "llmisvc-controller-manager-777c6bd7d5-l6mcc" (UID: "adac090f-b466-4ea0-b254-ec2d65dd63af") : secret "llmisvc-webhook-server-cert" not found Jul 07 15:47:39.607861 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:39.607836 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678k74l4"] Jul 07 15:47:39.608485 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:39.608463 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8a5048fd-5651-491f-9cea-7f4d93706202-cert\") pod \"kserve-controller-manager-d6f4b65b8-gz46p\" (UID: \"8a5048fd-5651-491f-9cea-7f4d93706202\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-gz46p" Jul 07 15:47:39.611681 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:39.611660 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678k74l4" Jul 07 15:47:39.615170 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:39.615149 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-xf4l9\"" Jul 07 15:47:39.617623 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:39.617598 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:47:39.619879 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:39.619860 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gfx8b\" (UniqueName: \"kubernetes.io/projected/adac090f-b466-4ea0-b254-ec2d65dd63af-kube-api-access-gfx8b\") pod \"llmisvc-controller-manager-777c6bd7d5-l6mcc\" (UID: \"adac090f-b466-4ea0-b254-ec2d65dd63af\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-l6mcc" Jul 07 15:47:39.621082 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:39.621062 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-phc89\" (UniqueName: \"kubernetes.io/projected/8a5048fd-5651-491f-9cea-7f4d93706202-kube-api-access-phc89\") pod \"kserve-controller-manager-d6f4b65b8-gz46p\" (UID: \"8a5048fd-5651-491f-9cea-7f4d93706202\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-gz46p" Jul 07 15:47:39.629546 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:39.629523 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678k74l4"] Jul 07 15:47:39.706189 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:39.706098 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5c36a68-d3a9-492c-a364-91b5785b16c1-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678k74l4\" (UID: \"a5c36a68-d3a9-492c-a364-91b5785b16c1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678k74l4" Jul 07 15:47:39.706189 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:39.706142 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5c36a68-d3a9-492c-a364-91b5785b16c1-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678k74l4\" (UID: \"a5c36a68-d3a9-492c-a364-91b5785b16c1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678k74l4" Jul 07 15:47:39.706189 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:39.706162 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5c36a68-d3a9-492c-a364-91b5785b16c1-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678k74l4\" (UID: \"a5c36a68-d3a9-492c-a364-91b5785b16c1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678k74l4" Jul 07 15:47:39.706189 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:39.706183 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5c36a68-d3a9-492c-a364-91b5785b16c1-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678k74l4\" (UID: \"a5c36a68-d3a9-492c-a364-91b5785b16c1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678k74l4" Jul 07 15:47:39.706466 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:39.706202 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5c36a68-d3a9-492c-a364-91b5785b16c1-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678k74l4\" (UID: \"a5c36a68-d3a9-492c-a364-91b5785b16c1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678k74l4" Jul 07 15:47:39.706466 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:39.706274 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5c36a68-d3a9-492c-a364-91b5785b16c1-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678k74l4\" (UID: \"a5c36a68-d3a9-492c-a364-91b5785b16c1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678k74l4" Jul 07 15:47:39.706466 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:39.706314 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5c36a68-d3a9-492c-a364-91b5785b16c1-istio-data\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678k74l4\" (UID: \"a5c36a68-d3a9-492c-a364-91b5785b16c1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678k74l4" Jul 07 15:47:39.706466 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:39.706355 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5c36a68-d3a9-492c-a364-91b5785b16c1-istio-token\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678k74l4\" (UID: \"a5c36a68-d3a9-492c-a364-91b5785b16c1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678k74l4" Jul 07 15:47:39.706466 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:39.706384 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wv7mp\" (UniqueName: \"kubernetes.io/projected/a5c36a68-d3a9-492c-a364-91b5785b16c1-kube-api-access-wv7mp\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678k74l4\" (UID: \"a5c36a68-d3a9-492c-a364-91b5785b16c1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678k74l4" Jul 07 15:47:39.739202 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:39.739171 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:47:39.758321 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:39.758284 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-gz46p" Jul 07 15:47:39.807198 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:39.807148 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5c36a68-d3a9-492c-a364-91b5785b16c1-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678k74l4\" (UID: \"a5c36a68-d3a9-492c-a364-91b5785b16c1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678k74l4" Jul 07 15:47:39.807342 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:39.807223 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5c36a68-d3a9-492c-a364-91b5785b16c1-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678k74l4\" (UID: \"a5c36a68-d3a9-492c-a364-91b5785b16c1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678k74l4" Jul 07 15:47:39.807342 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:39.807261 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5c36a68-d3a9-492c-a364-91b5785b16c1-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678k74l4\" (UID: \"a5c36a68-d3a9-492c-a364-91b5785b16c1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678k74l4" Jul 07 15:47:39.807615 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:39.807394 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5c36a68-d3a9-492c-a364-91b5785b16c1-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678k74l4\" (UID: \"a5c36a68-d3a9-492c-a364-91b5785b16c1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678k74l4" Jul 07 15:47:39.807615 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:39.807426 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5c36a68-d3a9-492c-a364-91b5785b16c1-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678k74l4\" (UID: \"a5c36a68-d3a9-492c-a364-91b5785b16c1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678k74l4" Jul 07 15:47:39.807615 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:39.807469 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5c36a68-d3a9-492c-a364-91b5785b16c1-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678k74l4\" (UID: \"a5c36a68-d3a9-492c-a364-91b5785b16c1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678k74l4" Jul 07 15:47:39.807615 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:39.807538 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5c36a68-d3a9-492c-a364-91b5785b16c1-istio-data\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678k74l4\" (UID: \"a5c36a68-d3a9-492c-a364-91b5785b16c1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678k74l4" Jul 07 15:47:39.807615 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:39.807609 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5c36a68-d3a9-492c-a364-91b5785b16c1-istio-token\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678k74l4\" (UID: \"a5c36a68-d3a9-492c-a364-91b5785b16c1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678k74l4" Jul 07 15:47:39.807863 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:39.807642 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5c36a68-d3a9-492c-a364-91b5785b16c1-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678k74l4\" (UID: \"a5c36a68-d3a9-492c-a364-91b5785b16c1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678k74l4" Jul 07 15:47:39.807863 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:39.807668 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wv7mp\" (UniqueName: \"kubernetes.io/projected/a5c36a68-d3a9-492c-a364-91b5785b16c1-kube-api-access-wv7mp\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678k74l4\" (UID: \"a5c36a68-d3a9-492c-a364-91b5785b16c1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678k74l4" Jul 07 15:47:39.808605 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:39.808419 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5c36a68-d3a9-492c-a364-91b5785b16c1-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678k74l4\" (UID: \"a5c36a68-d3a9-492c-a364-91b5785b16c1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678k74l4" Jul 07 15:47:39.808605 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:39.808445 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5c36a68-d3a9-492c-a364-91b5785b16c1-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678k74l4\" (UID: \"a5c36a68-d3a9-492c-a364-91b5785b16c1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678k74l4" Jul 07 15:47:39.808917 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:39.808895 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5c36a68-d3a9-492c-a364-91b5785b16c1-istio-data\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678k74l4\" (UID: \"a5c36a68-d3a9-492c-a364-91b5785b16c1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678k74l4" Jul 07 15:47:39.809086 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:39.809060 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5c36a68-d3a9-492c-a364-91b5785b16c1-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678k74l4\" (UID: \"a5c36a68-d3a9-492c-a364-91b5785b16c1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678k74l4" Jul 07 15:47:39.810458 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:39.810411 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5c36a68-d3a9-492c-a364-91b5785b16c1-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678k74l4\" (UID: \"a5c36a68-d3a9-492c-a364-91b5785b16c1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678k74l4" Jul 07 15:47:39.810701 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:39.810682 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5c36a68-d3a9-492c-a364-91b5785b16c1-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678k74l4\" (UID: \"a5c36a68-d3a9-492c-a364-91b5785b16c1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678k74l4" Jul 07 15:47:39.824473 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:39.824439 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5c36a68-d3a9-492c-a364-91b5785b16c1-istio-token\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678k74l4\" (UID: \"a5c36a68-d3a9-492c-a364-91b5785b16c1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678k74l4" Jul 07 15:47:39.824622 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:39.824578 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wv7mp\" (UniqueName: \"kubernetes.io/projected/a5c36a68-d3a9-492c-a364-91b5785b16c1-kube-api-access-wv7mp\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678k74l4\" (UID: \"a5c36a68-d3a9-492c-a364-91b5785b16c1\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678k74l4" Jul 07 15:47:39.909673 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:39.909646 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-gz46p"] Jul 07 15:47:39.911415 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:47:39.911380 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a5048fd_5651_491f_9cea_7f4d93706202.slice/crio-0ec8fb94d3147ed7270c543f361a7e305f5937bb1af400a203bcc5c63fe19a45 WatchSource:0}: Error finding container 0ec8fb94d3147ed7270c543f361a7e305f5937bb1af400a203bcc5c63fe19a45: Status 404 returned error can't find the container with id 0ec8fb94d3147ed7270c543f361a7e305f5937bb1af400a203bcc5c63fe19a45 Jul 07 15:47:39.940473 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:39.940445 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678k74l4" Jul 07 15:47:39.995629 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:39.994361 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:47:40.111730 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:40.111690 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/adac090f-b466-4ea0-b254-ec2d65dd63af-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-l6mcc\" (UID: \"adac090f-b466-4ea0-b254-ec2d65dd63af\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-l6mcc" Jul 07 15:47:40.114137 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:40.114118 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/adac090f-b466-4ea0-b254-ec2d65dd63af-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-l6mcc\" (UID: \"adac090f-b466-4ea0-b254-ec2d65dd63af\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-l6mcc" Jul 07 15:47:40.147074 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:40.147033 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678k74l4"] Jul 07 15:47:40.164734 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:47:40.164704 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5c36a68_d3a9_492c_a364_91b5785b16c1.slice/crio-827de8f0df166c4b71544bb06780765ef0111e5f2e0b477b653ca31e3de93f68 WatchSource:0}: Error finding container 827de8f0df166c4b71544bb06780765ef0111e5f2e0b477b653ca31e3de93f68: Status 404 returned error can't find the container with id 827de8f0df166c4b71544bb06780765ef0111e5f2e0b477b653ca31e3de93f68 Jul 07 15:47:40.191021 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:40.190996 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:47:40.271781 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:40.271698 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-gz46p" event={"ID":"8a5048fd-5651-491f-9cea-7f4d93706202","Type":"ContainerStarted","Data":"0ec8fb94d3147ed7270c543f361a7e305f5937bb1af400a203bcc5c63fe19a45"} Jul 07 15:47:40.272816 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:40.272792 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678k74l4" event={"ID":"a5c36a68-d3a9-492c-a364-91b5785b16c1","Type":"ContainerStarted","Data":"827de8f0df166c4b71544bb06780765ef0111e5f2e0b477b653ca31e3de93f68"} Jul 07 15:47:40.273547 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:40.273530 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-xmjjf" Jul 07 15:47:40.286533 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:40.286507 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-xmjjf"] Jul 07 15:47:40.350525 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:40.350484 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-l6mcc" Jul 07 15:47:40.504657 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:40.504625 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-l6mcc"] Jul 07 15:47:40.506888 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:47:40.506863 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podadac090f_b466_4ea0_b254_ec2d65dd63af.slice/crio-b91c9af4d3b3a25460874fa5d72e5d8e9b0f6dad59756dd279e88290c15b2317 WatchSource:0}: Error finding container b91c9af4d3b3a25460874fa5d72e5d8e9b0f6dad59756dd279e88290c15b2317: Status 404 returned error can't find the container with id b91c9af4d3b3a25460874fa5d72e5d8e9b0f6dad59756dd279e88290c15b2317 Jul 07 15:47:40.745910 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:40.745871 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:47:40.912097 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:40.912068 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:47:41.282120 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:41.282036 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-l6mcc" event={"ID":"adac090f-b466-4ea0-b254-ec2d65dd63af","Type":"ContainerStarted","Data":"b91c9af4d3b3a25460874fa5d72e5d8e9b0f6dad59756dd279e88290c15b2317"} Jul 07 15:47:42.231238 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:42.231208 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-4wkpn" Jul 07 15:47:42.287963 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:42.287921 2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7574459656-xmjjf" podUID="44256b42-3d30-4312-be20-ae4db826e66c" containerName="discovery" containerID="cri-o://05397023ac0329ff9a038bb23cfaf599b2e4258d14551e820aa939dff2f4e381" gracePeriod=30 Jul 07 15:47:43.274256 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:43.274223 2565 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7574459656-xmjjf container/discovery namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.21:8080/ready\": dial tcp 10.133.0.21:8080: connect: connection refused" start-of-body= Jul 07 15:47:43.274429 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:43.274284 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7574459656-xmjjf" podUID="44256b42-3d30-4312-be20-ae4db826e66c" containerName="discovery" probeResult="failure" output="Get \"http://10.133.0.21:8080/ready\": dial tcp 10.133.0.21:8080: connect: connection refused" Jul 07 15:47:43.294244 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:43.294217 2565 generic.go:356] "Generic (PLEG): container finished" podID="44256b42-3d30-4312-be20-ae4db826e66c" containerID="05397023ac0329ff9a038bb23cfaf599b2e4258d14551e820aa939dff2f4e381" exitCode=0 Jul 07 15:47:43.294659 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:43.294300 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-xmjjf" event={"ID":"44256b42-3d30-4312-be20-ae4db826e66c","Type":"ContainerDied","Data":"05397023ac0329ff9a038bb23cfaf599b2e4258d14551e820aa939dff2f4e381"} Jul 07 15:47:43.375085 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:43.375057 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-xmjjf" Jul 07 15:47:43.396717 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:43.396665 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 07 15:47:43.396874 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:43.396764 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 07 15:47:43.396874 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:43.396807 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 07 15:47:43.444405 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:43.444375 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44256b42-3d30-4312-be20-ae4db826e66c-istio-token\") pod \"44256b42-3d30-4312-be20-ae4db826e66c\" (UID: \"44256b42-3d30-4312-be20-ae4db826e66c\") " Jul 07 15:47:43.444580 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:43.444418 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/44256b42-3d30-4312-be20-ae4db826e66c-istio-csr-dns-cert\") pod \"44256b42-3d30-4312-be20-ae4db826e66c\" (UID: \"44256b42-3d30-4312-be20-ae4db826e66c\") " Jul 07 15:47:43.444580 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:43.444457 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/44256b42-3d30-4312-be20-ae4db826e66c-istio-csr-ca-configmap\") pod \"44256b42-3d30-4312-be20-ae4db826e66c\" (UID: \"44256b42-3d30-4312-be20-ae4db826e66c\") " Jul 07 15:47:43.444580 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:43.444503 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/44256b42-3d30-4312-be20-ae4db826e66c-istio-kubeconfig\") pod \"44256b42-3d30-4312-be20-ae4db826e66c\" (UID: \"44256b42-3d30-4312-be20-ae4db826e66c\") " Jul 07 15:47:43.444580 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:43.444576 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/44256b42-3d30-4312-be20-ae4db826e66c-local-certs\") pod \"44256b42-3d30-4312-be20-ae4db826e66c\" (UID: \"44256b42-3d30-4312-be20-ae4db826e66c\") " Jul 07 15:47:43.444795 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:43.444603 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/44256b42-3d30-4312-be20-ae4db826e66c-cacerts\") pod \"44256b42-3d30-4312-be20-ae4db826e66c\" (UID: \"44256b42-3d30-4312-be20-ae4db826e66c\") " Jul 07 15:47:43.444795 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:43.444628 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sb2xq\" (UniqueName: \"kubernetes.io/projected/44256b42-3d30-4312-be20-ae4db826e66c-kube-api-access-sb2xq\") pod \"44256b42-3d30-4312-be20-ae4db826e66c\" (UID: \"44256b42-3d30-4312-be20-ae4db826e66c\") " Jul 07 15:47:43.445446 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:43.445377 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44256b42-3d30-4312-be20-ae4db826e66c-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "44256b42-3d30-4312-be20-ae4db826e66c" (UID: "44256b42-3d30-4312-be20-ae4db826e66c"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 15:47:43.447297 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:43.447256 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/44256b42-3d30-4312-be20-ae4db826e66c-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "44256b42-3d30-4312-be20-ae4db826e66c" (UID: "44256b42-3d30-4312-be20-ae4db826e66c"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 15:47:43.447297 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:43.447274 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/44256b42-3d30-4312-be20-ae4db826e66c-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "44256b42-3d30-4312-be20-ae4db826e66c" (UID: "44256b42-3d30-4312-be20-ae4db826e66c"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 15:47:43.447475 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:43.447316 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44256b42-3d30-4312-be20-ae4db826e66c-kube-api-access-sb2xq" (OuterVolumeSpecName: "kube-api-access-sb2xq") pod "44256b42-3d30-4312-be20-ae4db826e66c" (UID: "44256b42-3d30-4312-be20-ae4db826e66c"). InnerVolumeSpecName "kube-api-access-sb2xq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 15:47:43.447766 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:43.447731 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/44256b42-3d30-4312-be20-ae4db826e66c-cacerts" (OuterVolumeSpecName: "cacerts") pod "44256b42-3d30-4312-be20-ae4db826e66c" (UID: "44256b42-3d30-4312-be20-ae4db826e66c"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 15:47:43.447892 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:43.447767 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44256b42-3d30-4312-be20-ae4db826e66c-local-certs" (OuterVolumeSpecName: "local-certs") pod "44256b42-3d30-4312-be20-ae4db826e66c" (UID: "44256b42-3d30-4312-be20-ae4db826e66c"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 15:47:43.448693 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:43.448667 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44256b42-3d30-4312-be20-ae4db826e66c-istio-token" (OuterVolumeSpecName: "istio-token") pod "44256b42-3d30-4312-be20-ae4db826e66c" (UID: "44256b42-3d30-4312-be20-ae4db826e66c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 15:47:43.545837 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:43.545798 2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44256b42-3d30-4312-be20-ae4db826e66c-istio-token\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 07 15:47:43.545837 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:43.545839 2565 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/44256b42-3d30-4312-be20-ae4db826e66c-istio-csr-dns-cert\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 07 15:47:43.546020 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:43.545857 2565 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/44256b42-3d30-4312-be20-ae4db826e66c-istio-csr-ca-configmap\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 07 15:47:43.546020 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:43.545873 2565 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/44256b42-3d30-4312-be20-ae4db826e66c-istio-kubeconfig\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 07 15:47:43.546020 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:43.545888 2565 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/44256b42-3d30-4312-be20-ae4db826e66c-local-certs\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 07 15:47:43.546020 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:43.545900 2565 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/44256b42-3d30-4312-be20-ae4db826e66c-cacerts\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 07 15:47:43.546020 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:43.545913 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sb2xq\" (UniqueName: \"kubernetes.io/projected/44256b42-3d30-4312-be20-ae4db826e66c-kube-api-access-sb2xq\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 07 15:47:44.242080 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:44.242053 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-q85t9" Jul 07 15:47:44.300414 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:44.300378 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678k74l4" event={"ID":"a5c36a68-d3a9-492c-a364-91b5785b16c1","Type":"ContainerStarted","Data":"bf4dcc9b30a969d08ed7f8f5f588109cb9342f04be144725c3075420bdffe099"} Jul 07 15:47:44.302105 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:44.302075 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-xmjjf" Jul 07 15:47:44.302105 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:44.302085 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-xmjjf" event={"ID":"44256b42-3d30-4312-be20-ae4db826e66c","Type":"ContainerDied","Data":"ecfdc1da3a14523478b0f87484f16ee04096518def3e82e1c282c317f92d1708"} Jul 07 15:47:44.302289 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:44.302138 2565 scope.go:117] "RemoveContainer" containerID="05397023ac0329ff9a038bb23cfaf599b2e4258d14551e820aa939dff2f4e381" Jul 07 15:47:44.304423 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:44.304372 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-l6mcc" event={"ID":"adac090f-b466-4ea0-b254-ec2d65dd63af","Type":"ContainerStarted","Data":"5b419b7e48332ea71c93e4fddadfc7aca0fe8927dc09534ae8ab6cd3ded16633"} Jul 07 15:47:44.304784 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:44.304765 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-l6mcc" Jul 07 15:47:44.308446 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:44.308061 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-gz46p" event={"ID":"8a5048fd-5651-491f-9cea-7f4d93706202","Type":"ContainerStarted","Data":"e4bb2fde2c10c458963ce01548ddd430ebf6991bb0ea74593de4ae0d9890f6b0"} Jul 07 15:47:44.308446 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:44.308210 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-gz46p" Jul 07 15:47:44.426898 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:44.426851 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678k74l4" podStartSLOduration=2.197328316 podStartE2EDuration="5.426833268s" totalImagesPullingTime="3.229504952s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:47:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:47:44.410767884 +0000 UTC m=+403.367345351" watchObservedRunningTime="2026-07-07 15:47:44.426833268 +0000 UTC m=+403.383410735" Jul 07 15:47:44.474478 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:44.474408 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-l6mcc" podStartSLOduration=1.865704177 podStartE2EDuration="5.47439048s" totalImagesPullingTime="3.608686303s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:47:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:47:44.472275279 +0000 UTC m=+403.428852747" watchObservedRunningTime="2026-07-07 15:47:44.47439048 +0000 UTC m=+403.430967951" Jul 07 15:47:44.531235 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:44.531171 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-gz46p" podStartSLOduration=2.090546376 podStartE2EDuration="5.531149331s" totalImagesPullingTime="3.440602955s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:47:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:47:44.530339158 +0000 UTC m=+403.486916627" watchObservedRunningTime="2026-07-07 15:47:44.531149331 +0000 UTC m=+403.487726803" Jul 07 15:47:44.560703 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:44.560666 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-xmjjf"] Jul 07 15:47:44.565077 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:44.565000 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-xmjjf"] Jul 07 15:47:44.941069 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:44.940993 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678k74l4" Jul 07 15:47:45.659599 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:45.659566 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="44256b42-3d30-4312-be20-ae4db826e66c" path="/var/lib/kubelet/pods/44256b42-3d30-4312-be20-ae4db826e66c/volumes" Jul 07 15:47:45.945848 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:45.945773 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678k74l4" Jul 07 15:47:46.316545 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:46.316510 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678k74l4" Jul 07 15:47:46.317405 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:47:46.317388 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678k74l4" Jul 07 15:48:15.315684 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:48:15.315649 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-l6mcc" Jul 07 15:48:15.318865 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:48:15.318840 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-gz46p" Jul 07 15:48:27.474105 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:48:27.474014 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:48:27.567547 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:48:27.567461 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:48:27.569285 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:48:27.569264 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-p7zch"] Jul 07 15:48:27.569613 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:48:27.569599 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="44256b42-3d30-4312-be20-ae4db826e66c" containerName="discovery" Jul 07 15:48:27.569677 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:48:27.569615 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="44256b42-3d30-4312-be20-ae4db826e66c" containerName="discovery" Jul 07 15:48:27.569677 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:48:27.569668 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="44256b42-3d30-4312-be20-ae4db826e66c" containerName="discovery" Jul 07 15:48:27.571352 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:48:27.571338 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-p7zch" Jul 07 15:48:27.574531 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:48:27.574509 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 07 15:48:27.574531 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:48:27.574516 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 07 15:48:27.575911 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:48:27.575796 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-h9hsw\"" Jul 07 15:48:27.583748 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:48:27.583723 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-p7zch"] Jul 07 15:48:27.610933 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:48:27.610898 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ftbhf\" (UniqueName: \"kubernetes.io/projected/34235094-2808-4c6a-814c-ec77369f4e55-kube-api-access-ftbhf\") pod \"kuadrant-operator-catalog-p7zch\" (UID: \"34235094-2808-4c6a-814c-ec77369f4e55\") " pod="kuadrant-system/kuadrant-operator-catalog-p7zch" Jul 07 15:48:27.651103 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:48:27.651072 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:48:27.711685 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:48:27.711654 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftbhf\" (UniqueName: \"kubernetes.io/projected/34235094-2808-4c6a-814c-ec77369f4e55-kube-api-access-ftbhf\") pod \"kuadrant-operator-catalog-p7zch\" (UID: \"34235094-2808-4c6a-814c-ec77369f4e55\") " pod="kuadrant-system/kuadrant-operator-catalog-p7zch" Jul 07 15:48:27.720850 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:48:27.720828 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftbhf\" (UniqueName: \"kubernetes.io/projected/34235094-2808-4c6a-814c-ec77369f4e55-kube-api-access-ftbhf\") pod \"kuadrant-operator-catalog-p7zch\" (UID: \"34235094-2808-4c6a-814c-ec77369f4e55\") " pod="kuadrant-system/kuadrant-operator-catalog-p7zch" Jul 07 15:48:27.881239 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:48:27.881197 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-p7zch" Jul 07 15:48:27.926410 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:48:27.926375 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-p7zch"] Jul 07 15:48:28.014298 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:48:28.014267 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-p7zch"] Jul 07 15:48:28.017209 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:48:28.017180 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34235094_2808_4c6a_814c_ec77369f4e55.slice/crio-138fd5307b96b860847aa445a817e31fae80fd6e51eb3a8adf1026d184305822 WatchSource:0}: Error finding container 138fd5307b96b860847aa445a817e31fae80fd6e51eb3a8adf1026d184305822: Status 404 returned error can't find the container with id 138fd5307b96b860847aa445a817e31fae80fd6e51eb3a8adf1026d184305822 Jul 07 15:48:28.135140 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:48:28.135068 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-6fszm"] Jul 07 15:48:28.137550 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:48:28.137534 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-6fszm" Jul 07 15:48:28.148828 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:48:28.148803 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-6fszm"] Jul 07 15:48:28.216452 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:48:28.216414 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7hvf\" (UniqueName: \"kubernetes.io/projected/62427603-a6ea-4c7d-98ee-347671a9d227-kube-api-access-k7hvf\") pod \"kuadrant-operator-catalog-6fszm\" (UID: \"62427603-a6ea-4c7d-98ee-347671a9d227\") " pod="kuadrant-system/kuadrant-operator-catalog-6fszm" Jul 07 15:48:28.317568 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:48:28.317527 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7hvf\" (UniqueName: \"kubernetes.io/projected/62427603-a6ea-4c7d-98ee-347671a9d227-kube-api-access-k7hvf\") pod \"kuadrant-operator-catalog-6fszm\" (UID: \"62427603-a6ea-4c7d-98ee-347671a9d227\") " pod="kuadrant-system/kuadrant-operator-catalog-6fszm" Jul 07 15:48:28.329429 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:48:28.329392 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7hvf\" (UniqueName: \"kubernetes.io/projected/62427603-a6ea-4c7d-98ee-347671a9d227-kube-api-access-k7hvf\") pod \"kuadrant-operator-catalog-6fszm\" (UID: \"62427603-a6ea-4c7d-98ee-347671a9d227\") " pod="kuadrant-system/kuadrant-operator-catalog-6fszm" Jul 07 15:48:28.446743 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:48:28.446652 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-6fszm" Jul 07 15:48:28.491038 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:48:28.491002 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-p7zch" event={"ID":"34235094-2808-4c6a-814c-ec77369f4e55","Type":"ContainerStarted","Data":"138fd5307b96b860847aa445a817e31fae80fd6e51eb3a8adf1026d184305822"} Jul 07 15:48:28.592236 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:48:28.592202 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-6fszm"] Jul 07 15:48:28.675356 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:48:28.675323 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62427603_a6ea_4c7d_98ee_347671a9d227.slice/crio-293cdbc7259a3ee5e625e688b37a0d97b5bb6ec16af12fbe4481da635c1bd5ff WatchSource:0}: Error finding container 293cdbc7259a3ee5e625e688b37a0d97b5bb6ec16af12fbe4481da635c1bd5ff: Status 404 returned error can't find the container with id 293cdbc7259a3ee5e625e688b37a0d97b5bb6ec16af12fbe4481da635c1bd5ff Jul 07 15:48:29.497445 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:48:29.497409 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-6fszm" event={"ID":"62427603-a6ea-4c7d-98ee-347671a9d227","Type":"ContainerStarted","Data":"293cdbc7259a3ee5e625e688b37a0d97b5bb6ec16af12fbe4481da635c1bd5ff"} Jul 07 15:48:30.503140 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:48:30.503097 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-p7zch" event={"ID":"34235094-2808-4c6a-814c-ec77369f4e55","Type":"ContainerStarted","Data":"1817e97cdda9c36676e1d92362462d36cc755663e0933ddb27ddcc12741293e5"} Jul 07 15:48:30.503140 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:48:30.503117 2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-p7zch" podUID="34235094-2808-4c6a-814c-ec77369f4e55" containerName="registry-server" containerID="cri-o://1817e97cdda9c36676e1d92362462d36cc755663e0933ddb27ddcc12741293e5" gracePeriod=2 Jul 07 15:48:30.504756 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:48:30.504734 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-6fszm" event={"ID":"62427603-a6ea-4c7d-98ee-347671a9d227","Type":"ContainerStarted","Data":"a86acd8aa2b8cf9f2dfabc5ceb8d982667640ad122766f0cc366fd1ad1fa1dcc"} Jul 07 15:48:30.542641 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:48:30.542587 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-p7zch" podStartSLOduration=1.7208505459999999 podStartE2EDuration="3.542564296s" totalImagesPullingTime="1.82171375s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:48:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:48:30.542079779 +0000 UTC m=+449.498657247" watchObservedRunningTime="2026-07-07 15:48:30.542564296 +0000 UTC m=+449.499141765" Jul 07 15:48:30.567270 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:48:30.567210 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-6fszm" podStartSLOduration=1.403508531 podStartE2EDuration="2.567183699s" totalImagesPullingTime="1.163675168s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:48:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:48:30.565031371 +0000 UTC m=+449.521608838" watchObservedRunningTime="2026-07-07 15:48:30.567183699 +0000 UTC m=+449.523761167" Jul 07 15:48:30.743245 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:48:30.743216 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-p7zch" Jul 07 15:48:30.841595 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:48:30.841560 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ftbhf\" (UniqueName: \"kubernetes.io/projected/34235094-2808-4c6a-814c-ec77369f4e55-kube-api-access-ftbhf\") pod \"34235094-2808-4c6a-814c-ec77369f4e55\" (UID: \"34235094-2808-4c6a-814c-ec77369f4e55\") " Jul 07 15:48:30.843724 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:48:30.843703 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34235094-2808-4c6a-814c-ec77369f4e55-kube-api-access-ftbhf" (OuterVolumeSpecName: "kube-api-access-ftbhf") pod "34235094-2808-4c6a-814c-ec77369f4e55" (UID: "34235094-2808-4c6a-814c-ec77369f4e55"). InnerVolumeSpecName "kube-api-access-ftbhf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 15:48:30.942129 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:48:30.942098 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ftbhf\" (UniqueName: \"kubernetes.io/projected/34235094-2808-4c6a-814c-ec77369f4e55-kube-api-access-ftbhf\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 07 15:48:31.510770 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:48:31.510730 2565 generic.go:356] "Generic (PLEG): container finished" podID="34235094-2808-4c6a-814c-ec77369f4e55" containerID="1817e97cdda9c36676e1d92362462d36cc755663e0933ddb27ddcc12741293e5" exitCode=0 Jul 07 15:48:31.511242 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:48:31.510794 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-p7zch" Jul 07 15:48:31.511242 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:48:31.510812 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-p7zch" event={"ID":"34235094-2808-4c6a-814c-ec77369f4e55","Type":"ContainerDied","Data":"1817e97cdda9c36676e1d92362462d36cc755663e0933ddb27ddcc12741293e5"} Jul 07 15:48:31.511242 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:48:31.510884 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-p7zch" event={"ID":"34235094-2808-4c6a-814c-ec77369f4e55","Type":"ContainerDied","Data":"138fd5307b96b860847aa445a817e31fae80fd6e51eb3a8adf1026d184305822"} Jul 07 15:48:31.511242 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:48:31.510906 2565 scope.go:117] "RemoveContainer" containerID="1817e97cdda9c36676e1d92362462d36cc755663e0933ddb27ddcc12741293e5" Jul 07 15:48:31.519920 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:48:31.519904 2565 scope.go:117] "RemoveContainer" containerID="1817e97cdda9c36676e1d92362462d36cc755663e0933ddb27ddcc12741293e5" Jul 07 15:48:31.520195 ip-10-0-132-65 kubenswrapper[2565]: E0707 15:48:31.520176 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1817e97cdda9c36676e1d92362462d36cc755663e0933ddb27ddcc12741293e5\": container with ID starting with 1817e97cdda9c36676e1d92362462d36cc755663e0933ddb27ddcc12741293e5 not found: ID does not exist" containerID="1817e97cdda9c36676e1d92362462d36cc755663e0933ddb27ddcc12741293e5" Jul 07 15:48:31.520262 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:48:31.520207 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1817e97cdda9c36676e1d92362462d36cc755663e0933ddb27ddcc12741293e5"} err="failed to get container status \"1817e97cdda9c36676e1d92362462d36cc755663e0933ddb27ddcc12741293e5\": rpc error: code = NotFound desc = could not find container \"1817e97cdda9c36676e1d92362462d36cc755663e0933ddb27ddcc12741293e5\": container with ID starting with 1817e97cdda9c36676e1d92362462d36cc755663e0933ddb27ddcc12741293e5 not found: ID does not exist" Jul 07 15:48:31.535062 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:48:31.535033 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-p7zch"] Jul 07 15:48:31.539544 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:48:31.539519 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-p7zch"] Jul 07 15:48:31.660394 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:48:31.660364 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="34235094-2808-4c6a-814c-ec77369f4e55" path="/var/lib/kubelet/pods/34235094-2808-4c6a-814c-ec77369f4e55/volumes" Jul 07 15:48:38.446790 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:48:38.446754 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-6fszm" Jul 07 15:48:38.447265 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:48:38.446821 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-6fszm" Jul 07 15:48:38.469027 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:48:38.469003 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-6fszm" Jul 07 15:48:38.564713 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:48:38.564684 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-6fszm" Jul 07 15:48:56.158778 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:48:56.158744 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:48:59.635317 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:48:59.635282 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:49:00.876558 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:49:00.876517 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2j52h"] Jul 07 15:49:00.877034 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:49:00.876841 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="34235094-2808-4c6a-814c-ec77369f4e55" containerName="registry-server" Jul 07 15:49:00.877034 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:49:00.876853 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="34235094-2808-4c6a-814c-ec77369f4e55" containerName="registry-server" Jul 07 15:49:00.877034 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:49:00.876913 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="34235094-2808-4c6a-814c-ec77369f4e55" containerName="registry-server" Jul 07 15:49:00.881790 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:49:00.881770 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2j52h" Jul 07 15:49:00.885209 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:49:00.885184 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-dxbl6\"" Jul 07 15:49:00.894965 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:49:00.894943 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2j52h"] Jul 07 15:49:01.009507 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:49:01.009449 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vxr2f\" (UniqueName: \"kubernetes.io/projected/377033ac-01ab-4e31-8953-63b3ca057da4-kube-api-access-vxr2f\") pod \"kuadrant-operator-controller-manager-8684c9ff49-2j52h\" (UID: \"377033ac-01ab-4e31-8953-63b3ca057da4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2j52h" Jul 07 15:49:01.009684 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:49:01.009537 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/377033ac-01ab-4e31-8953-63b3ca057da4-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-2j52h\" (UID: \"377033ac-01ab-4e31-8953-63b3ca057da4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2j52h" Jul 07 15:49:01.110230 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:49:01.110195 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/377033ac-01ab-4e31-8953-63b3ca057da4-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-2j52h\" (UID: \"377033ac-01ab-4e31-8953-63b3ca057da4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2j52h" Jul 07 15:49:01.110416 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:49:01.110296 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vxr2f\" (UniqueName: \"kubernetes.io/projected/377033ac-01ab-4e31-8953-63b3ca057da4-kube-api-access-vxr2f\") pod \"kuadrant-operator-controller-manager-8684c9ff49-2j52h\" (UID: \"377033ac-01ab-4e31-8953-63b3ca057da4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2j52h" Jul 07 15:49:01.110659 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:49:01.110636 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/377033ac-01ab-4e31-8953-63b3ca057da4-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-2j52h\" (UID: \"377033ac-01ab-4e31-8953-63b3ca057da4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2j52h" Jul 07 15:49:01.120564 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:49:01.120541 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vxr2f\" (UniqueName: \"kubernetes.io/projected/377033ac-01ab-4e31-8953-63b3ca057da4-kube-api-access-vxr2f\") pod \"kuadrant-operator-controller-manager-8684c9ff49-2j52h\" (UID: \"377033ac-01ab-4e31-8953-63b3ca057da4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2j52h" Jul 07 15:49:01.191645 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:49:01.191553 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2j52h" Jul 07 15:49:01.335337 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:49:01.335315 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2j52h"] Jul 07 15:49:01.337865 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:49:01.337839 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod377033ac_01ab_4e31_8953_63b3ca057da4.slice/crio-d9163f7ad22c818eefc7d8d6a01ceacfa21252ab0805d1dd5766741256f46022 WatchSource:0}: Error finding container d9163f7ad22c818eefc7d8d6a01ceacfa21252ab0805d1dd5766741256f46022: Status 404 returned error can't find the container with id d9163f7ad22c818eefc7d8d6a01ceacfa21252ab0805d1dd5766741256f46022 Jul 07 15:49:01.564061 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:49:01.564029 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:49:01.645887 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:49:01.645845 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2j52h" event={"ID":"377033ac-01ab-4e31-8953-63b3ca057da4","Type":"ContainerStarted","Data":"d9163f7ad22c818eefc7d8d6a01ceacfa21252ab0805d1dd5766741256f46022"} Jul 07 15:49:02.857639 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:49:02.857610 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-m8d5b"] Jul 07 15:49:02.861628 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:49:02.861609 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-m8d5b" Jul 07 15:49:02.864591 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:49:02.864566 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Jul 07 15:49:02.864720 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:49:02.864566 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-94mxg\"" Jul 07 15:49:02.872712 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:49:02.872690 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-m8d5b"] Jul 07 15:49:03.028648 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:49:03.028615 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fntk5\" (UniqueName: \"kubernetes.io/projected/2c9e0255-074e-4dcc-a8e8-ae664d60d2bb-kube-api-access-fntk5\") pod \"dns-operator-controller-manager-864dd78c4c-m8d5b\" (UID: \"2c9e0255-074e-4dcc-a8e8-ae664d60d2bb\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-m8d5b" Jul 07 15:49:03.129204 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:49:03.129117 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fntk5\" (UniqueName: \"kubernetes.io/projected/2c9e0255-074e-4dcc-a8e8-ae664d60d2bb-kube-api-access-fntk5\") pod \"dns-operator-controller-manager-864dd78c4c-m8d5b\" (UID: \"2c9e0255-074e-4dcc-a8e8-ae664d60d2bb\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-m8d5b" Jul 07 15:49:03.143595 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:49:03.143561 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fntk5\" (UniqueName: \"kubernetes.io/projected/2c9e0255-074e-4dcc-a8e8-ae664d60d2bb-kube-api-access-fntk5\") pod \"dns-operator-controller-manager-864dd78c4c-m8d5b\" (UID: \"2c9e0255-074e-4dcc-a8e8-ae664d60d2bb\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-m8d5b" Jul 07 15:49:03.173423 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:49:03.173393 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-m8d5b" Jul 07 15:49:03.325478 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:49:03.325454 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-m8d5b"] Jul 07 15:49:03.327790 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:49:03.327762 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c9e0255_074e_4dcc_a8e8_ae664d60d2bb.slice/crio-d063183a324ddc6e1883bb8ebabbabff10dffc7fdadf12f932464189a8e44c4e WatchSource:0}: Error finding container d063183a324ddc6e1883bb8ebabbabff10dffc7fdadf12f932464189a8e44c4e: Status 404 returned error can't find the container with id d063183a324ddc6e1883bb8ebabbabff10dffc7fdadf12f932464189a8e44c4e Jul 07 15:49:03.401037 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:49:03.400952 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:49:03.558064 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:49:03.558018 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:49:03.659759 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:49:03.659680 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-m8d5b" event={"ID":"2c9e0255-074e-4dcc-a8e8-ae664d60d2bb","Type":"ContainerStarted","Data":"d063183a324ddc6e1883bb8ebabbabff10dffc7fdadf12f932464189a8e44c4e"} Jul 07 15:49:05.667272 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:49:05.667239 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2j52h" event={"ID":"377033ac-01ab-4e31-8953-63b3ca057da4","Type":"ContainerStarted","Data":"7d9d4e8c1fd73c44b20cf7d083b31af1f05665112c570a82e57af20b0e31f33f"} Jul 07 15:49:05.667673 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:49:05.667370 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2j52h" Jul 07 15:49:05.693085 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:49:05.693007 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2j52h" podStartSLOduration=2.041981124 podStartE2EDuration="5.69298605s" totalImagesPullingTime="3.651004926s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:49:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:49:05.69007642 +0000 UTC m=+484.646653884" watchObservedRunningTime="2026-07-07 15:49:05.69298605 +0000 UTC m=+484.649563518" Jul 07 15:49:05.906126 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:49:05.906087 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-s6jjq"] Jul 07 15:49:05.909573 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:49:05.909550 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-s6jjq" Jul 07 15:49:05.912380 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:49:05.912359 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-zxgj7\"" Jul 07 15:49:05.922116 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:49:05.922058 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-s6jjq"] Jul 07 15:49:05.994282 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:49:05.994243 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:49:06.056963 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:49:06.056931 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kh6wb\" (UniqueName: \"kubernetes.io/projected/accc5140-f794-42b1-a0a1-6141964184da-kube-api-access-kh6wb\") pod \"limitador-operator-controller-manager-859d7b78bd-s6jjq\" (UID: \"accc5140-f794-42b1-a0a1-6141964184da\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-s6jjq" Jul 07 15:49:06.157678 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:49:06.157637 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kh6wb\" (UniqueName: \"kubernetes.io/projected/accc5140-f794-42b1-a0a1-6141964184da-kube-api-access-kh6wb\") pod \"limitador-operator-controller-manager-859d7b78bd-s6jjq\" (UID: \"accc5140-f794-42b1-a0a1-6141964184da\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-s6jjq" Jul 07 15:49:06.169176 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:49:06.169141 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kh6wb\" (UniqueName: \"kubernetes.io/projected/accc5140-f794-42b1-a0a1-6141964184da-kube-api-access-kh6wb\") pod \"limitador-operator-controller-manager-859d7b78bd-s6jjq\" (UID: \"accc5140-f794-42b1-a0a1-6141964184da\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-s6jjq" Jul 07 15:49:06.221577 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:49:06.221483 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-s6jjq" Jul 07 15:49:06.362379 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:49:06.362348 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:49:06.378190 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:49:06.378164 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-s6jjq"] Jul 07 15:49:06.380834 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:49:06.380806 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaccc5140_f794_42b1_a0a1_6141964184da.slice/crio-ab6d6b5967e82b025920e0a217f0c782788a8ae9e7ff6070ab5e0133f0898436 WatchSource:0}: Error finding container ab6d6b5967e82b025920e0a217f0c782788a8ae9e7ff6070ab5e0133f0898436: Status 404 returned error can't find the container with id ab6d6b5967e82b025920e0a217f0c782788a8ae9e7ff6070ab5e0133f0898436 Jul 07 15:49:06.610113 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:49:06.610075 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:49:06.674276 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:49:06.674238 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-m8d5b" event={"ID":"2c9e0255-074e-4dcc-a8e8-ae664d60d2bb","Type":"ContainerStarted","Data":"48f3a40a9530f63fc757490ccec5e6af37ffce867151989ef0fa24ca76cc368f"} Jul 07 15:49:06.674729 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:49:06.674365 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-m8d5b" Jul 07 15:49:06.675559 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:49:06.675534 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-s6jjq" event={"ID":"accc5140-f794-42b1-a0a1-6141964184da","Type":"ContainerStarted","Data":"ab6d6b5967e82b025920e0a217f0c782788a8ae9e7ff6070ab5e0133f0898436"} Jul 07 15:49:06.722511 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:49:06.720342 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-m8d5b" podStartSLOduration=1.764604808 podStartE2EDuration="4.720316623s" totalImagesPullingTime="2.955711815s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:49:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:49:06.717866725 +0000 UTC m=+485.674444251" watchObservedRunningTime="2026-07-07 15:49:06.720316623 +0000 UTC m=+485.676894091" Jul 07 15:49:07.377606 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:49:07.377567 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:49:08.037307 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:49:08.037222 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:49:08.143051 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:49:08.143015 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:49:08.688250 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:49:08.688213 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-s6jjq" event={"ID":"accc5140-f794-42b1-a0a1-6141964184da","Type":"ContainerStarted","Data":"cd07a1015b845d4fac450b29ee9babb09d8b21965318c798b54b6659816404df"} Jul 07 15:49:08.688437 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:49:08.688342 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-s6jjq" Jul 07 15:49:08.709958 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:49:08.709913 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-s6jjq" podStartSLOduration=2.335335414 podStartE2EDuration="3.70989751s" totalImagesPullingTime="1.374562096s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:49:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:49:08.709662367 +0000 UTC m=+487.666239835" watchObservedRunningTime="2026-07-07 15:49:08.70989751 +0000 UTC m=+487.666474977" Jul 07 15:49:10.398379 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:49:10.398345 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-wvkvz"] Jul 07 15:49:10.402062 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:49:10.402042 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-wvkvz" Jul 07 15:49:10.404639 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:49:10.404620 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-nvrrv\"" Jul 07 15:49:10.411769 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:49:10.411747 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-wvkvz"] Jul 07 15:49:10.598297 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:49:10.598265 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzx7d\" (UniqueName: \"kubernetes.io/projected/6957a016-f9f9-4ec4-bc6a-1e8edc3ed6fa-kube-api-access-bzx7d\") pod \"authorino-operator-75cc8d5cb7-wvkvz\" (UID: \"6957a016-f9f9-4ec4-bc6a-1e8edc3ed6fa\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-wvkvz" Jul 07 15:49:10.699722 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:49:10.699637 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzx7d\" (UniqueName: \"kubernetes.io/projected/6957a016-f9f9-4ec4-bc6a-1e8edc3ed6fa-kube-api-access-bzx7d\") pod \"authorino-operator-75cc8d5cb7-wvkvz\" (UID: \"6957a016-f9f9-4ec4-bc6a-1e8edc3ed6fa\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-wvkvz" Jul 07 15:49:10.709900 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:49:10.709871 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzx7d\" (UniqueName: \"kubernetes.io/projected/6957a016-f9f9-4ec4-bc6a-1e8edc3ed6fa-kube-api-access-bzx7d\") pod \"authorino-operator-75cc8d5cb7-wvkvz\" (UID: \"6957a016-f9f9-4ec4-bc6a-1e8edc3ed6fa\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-wvkvz" Jul 07 15:49:10.711662 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:49:10.711644 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-wvkvz" Jul 07 15:49:10.851059 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:49:10.851026 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-wvkvz"] Jul 07 15:49:10.854589 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:49:10.854565 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6957a016_f9f9_4ec4_bc6a_1e8edc3ed6fa.slice/crio-30cc96ae87151044d3713088e16e326aee4715a4f49e146e3aee25901d818154 WatchSource:0}: Error finding container 30cc96ae87151044d3713088e16e326aee4715a4f49e146e3aee25901d818154: Status 404 returned error can't find the container with id 30cc96ae87151044d3713088e16e326aee4715a4f49e146e3aee25901d818154 Jul 07 15:49:11.704564 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:49:11.704529 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-wvkvz" event={"ID":"6957a016-f9f9-4ec4-bc6a-1e8edc3ed6fa","Type":"ContainerStarted","Data":"30cc96ae87151044d3713088e16e326aee4715a4f49e146e3aee25901d818154"} Jul 07 15:49:13.368929 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:49:13.368894 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:49:13.469163 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:49:13.469122 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:49:13.713872 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:49:13.713781 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-wvkvz" event={"ID":"6957a016-f9f9-4ec4-bc6a-1e8edc3ed6fa","Type":"ContainerStarted","Data":"3914bbe65f386305c0a98237fb1e5ad1da9bc4ddc76551135b48529b384f523b"} Jul 07 15:49:13.714025 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:49:13.713891 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-wvkvz" Jul 07 15:49:13.740024 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:49:13.739975 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-75cc8d5cb7-wvkvz" podStartSLOduration=1.596739095 podStartE2EDuration="3.739960944s" totalImagesPullingTime="2.143221849s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:49:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:49:13.738803706 +0000 UTC m=+492.695381184" watchObservedRunningTime="2026-07-07 15:49:13.739960944 +0000 UTC m=+492.696538453" Jul 07 15:49:14.787032 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:49:14.787004 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:49:14.906325 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:49:14.906290 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:49:16.678019 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:49:16.677992 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2j52h" Jul 07 15:49:17.105825 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:49:17.105790 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:49:17.232921 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:49:17.232889 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:49:17.683942 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:49:17.683910 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-m8d5b" Jul 07 15:49:18.554407 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:49:18.554373 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2j52h"] Jul 07 15:49:18.554666 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:49:18.554626 2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2j52h" podUID="377033ac-01ab-4e31-8953-63b3ca057da4" containerName="manager" containerID="cri-o://7d9d4e8c1fd73c44b20cf7d083b31af1f05665112c570a82e57af20b0e31f33f" gracePeriod=2 Jul 07 15:49:18.559702 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:49:18.559677 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2j52h"] Jul 07 15:49:18.573172 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:49:18.573143 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-s6jjq"] Jul 07 15:49:18.573395 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:49:18.573365 2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-s6jjq" podUID="accc5140-f794-42b1-a0a1-6141964184da" containerName="manager" containerID="cri-o://cd07a1015b845d4fac450b29ee9babb09d8b21965318c798b54b6659816404df" gracePeriod=2 Jul 07 15:49:18.575607 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:49:18.575573 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-s6jjq" Jul 07 15:49:18.576439 ip-10-0-132-65 kubenswrapper[2565]: E0707 15:49:18.576408 2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-2j52h\" is forbidden: User \"system:node:ip-10-0-132-65.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-65.ec2.internal' and this object" podUID="377033ac-01ab-4e31-8953-63b3ca057da4" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2j52h" Jul 07 15:49:18.579255 ip-10-0-132-65 kubenswrapper[2565]: E0707 15:49:18.579223 2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-s6jjq\" is forbidden: User \"system:node:ip-10-0-132-65.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-65.ec2.internal' and this object" podUID="accc5140-f794-42b1-a0a1-6141964184da" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-s6jjq" Jul 07 15:49:18.581414 ip-10-0-132-65 kubenswrapper[2565]: E0707 15:49:18.581355 2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-2j52h\" is forbidden: User \"system:node:ip-10-0-132-65.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-65.ec2.internal' and this object" podUID="377033ac-01ab-4e31-8953-63b3ca057da4" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2j52h" Jul 07 15:49:18.581733 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:49:18.581681 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wj6jx"] Jul 07 15:49:18.582593 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:49:18.582269 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="377033ac-01ab-4e31-8953-63b3ca057da4" containerName="manager" Jul 07 15:49:18.582593 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:49:18.582291 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="377033ac-01ab-4e31-8953-63b3ca057da4" containerName="manager" Jul 07 15:49:18.582593 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:49:18.582365 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="377033ac-01ab-4e31-8953-63b3ca057da4" containerName="manager" Jul 07 15:49:18.598379 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:49:18.598314 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-s6jjq"] Jul 07 15:49:18.598746 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:49:18.598682 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wj6jx" Jul 07 15:49:18.613709 ip-10-0-132-65 kubenswrapper[2565]: E0707 15:49:18.613639 2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-s6jjq\" is forbidden: User \"system:node:ip-10-0-132-65.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-65.ec2.internal' and this object" podUID="accc5140-f794-42b1-a0a1-6141964184da" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-s6jjq" Jul 07 15:49:18.615573 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:49:18.615548 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-nx6zw"] Jul 07 15:49:18.616047 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:49:18.616029 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="accc5140-f794-42b1-a0a1-6141964184da" containerName="manager" Jul 07 15:49:18.616124 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:49:18.616048 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="accc5140-f794-42b1-a0a1-6141964184da" containerName="manager" Jul 07 15:49:18.616190 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:49:18.616128 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="accc5140-f794-42b1-a0a1-6141964184da" containerName="manager" Jul 07 15:49:18.616786 ip-10-0-132-65 kubenswrapper[2565]: E0707 15:49:18.616647 2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-2j52h\" is forbidden: User \"system:node:ip-10-0-132-65.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-65.ec2.internal' and this object" podUID="377033ac-01ab-4e31-8953-63b3ca057da4" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2j52h" Jul 07 15:49:18.619244 ip-10-0-132-65 kubenswrapper[2565]: E0707 15:49:18.619209 2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-s6jjq\" is forbidden: User \"system:node:ip-10-0-132-65.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-65.ec2.internal' and this object" podUID="accc5140-f794-42b1-a0a1-6141964184da" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-s6jjq" Jul 07 15:49:18.619382 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:49:18.619355 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wj6jx"] Jul 07 15:49:18.619462 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:49:18.619420 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-nx6zw" Jul 07 15:49:18.619845 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:49:18.619823 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-nx6zw"] Jul 07 15:49:18.641690 ip-10-0-132-65 kubenswrapper[2565]: E0707 15:49:18.641656 2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-2j52h\" is forbidden: User \"system:node:ip-10-0-132-65.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-65.ec2.internal' and this object" podUID="377033ac-01ab-4e31-8953-63b3ca057da4" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2j52h" Jul 07 15:49:18.643933 ip-10-0-132-65 kubenswrapper[2565]: E0707 15:49:18.643908 2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-s6jjq\" is forbidden: User \"system:node:ip-10-0-132-65.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-65.ec2.internal' and this object" podUID="accc5140-f794-42b1-a0a1-6141964184da" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-s6jjq" Jul 07 15:49:18.659910 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:49:18.659877 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/395e1cf5-09da-4d48-aa8f-bc91863884ec-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-wj6jx\" (UID: \"395e1cf5-09da-4d48-aa8f-bc91863884ec\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wj6jx" Jul 07 15:49:18.660022 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:49:18.659950 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s55cq\" (UniqueName: \"kubernetes.io/projected/6555056e-fa42-4326-becf-9bd8eb0cfc21-kube-api-access-s55cq\") pod \"limitador-operator-controller-manager-859d7b78bd-nx6zw\" (UID: \"6555056e-fa42-4326-becf-9bd8eb0cfc21\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-nx6zw" Jul 07 15:49:18.660022 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:49:18.659980 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8tbj5\" (UniqueName: \"kubernetes.io/projected/395e1cf5-09da-4d48-aa8f-bc91863884ec-kube-api-access-8tbj5\") pod \"kuadrant-operator-controller-manager-8684c9ff49-wj6jx\" (UID: \"395e1cf5-09da-4d48-aa8f-bc91863884ec\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wj6jx" Jul 07 15:49:18.741291 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:49:18.741242 2565 generic.go:356] "Generic (PLEG): container finished" podID="accc5140-f794-42b1-a0a1-6141964184da" containerID="cd07a1015b845d4fac450b29ee9babb09d8b21965318c798b54b6659816404df" exitCode=0 Jul 07 15:49:18.744182 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:49:18.744099 2565 generic.go:356] "Generic (PLEG): container finished" podID="377033ac-01ab-4e31-8953-63b3ca057da4" containerID="7d9d4e8c1fd73c44b20cf7d083b31af1f05665112c570a82e57af20b0e31f33f" exitCode=0 Jul 07 15:49:18.761096 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:49:18.761060 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s55cq\" (UniqueName: \"kubernetes.io/projected/6555056e-fa42-4326-becf-9bd8eb0cfc21-kube-api-access-s55cq\") pod \"limitador-operator-controller-manager-859d7b78bd-nx6zw\" (UID: \"6555056e-fa42-4326-becf-9bd8eb0cfc21\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-nx6zw" Jul 07 15:49:18.761245 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:49:18.761115 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8tbj5\" (UniqueName: \"kubernetes.io/projected/395e1cf5-09da-4d48-aa8f-bc91863884ec-kube-api-access-8tbj5\") pod \"kuadrant-operator-controller-manager-8684c9ff49-wj6jx\" (UID: \"395e1cf5-09da-4d48-aa8f-bc91863884ec\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wj6jx" Jul 07 15:49:18.761245 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:49:18.761206 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/395e1cf5-09da-4d48-aa8f-bc91863884ec-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-wj6jx\" (UID: \"395e1cf5-09da-4d48-aa8f-bc91863884ec\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wj6jx" Jul 07 15:49:18.761613 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:49:18.761592 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/395e1cf5-09da-4d48-aa8f-bc91863884ec-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-wj6jx\" (UID: \"395e1cf5-09da-4d48-aa8f-bc91863884ec\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wj6jx" Jul 07 15:49:18.771521 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:49:18.771463 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8tbj5\" (UniqueName: \"kubernetes.io/projected/395e1cf5-09da-4d48-aa8f-bc91863884ec-kube-api-access-8tbj5\") pod \"kuadrant-operator-controller-manager-8684c9ff49-wj6jx\" (UID: \"395e1cf5-09da-4d48-aa8f-bc91863884ec\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wj6jx" Jul 07 15:49:18.771676 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:49:18.771654 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s55cq\" (UniqueName: \"kubernetes.io/projected/6555056e-fa42-4326-becf-9bd8eb0cfc21-kube-api-access-s55cq\") pod \"limitador-operator-controller-manager-859d7b78bd-nx6zw\" (UID: \"6555056e-fa42-4326-becf-9bd8eb0cfc21\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-nx6zw" Jul 07 15:49:18.809596 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:49:18.809577 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2j52h" Jul 07 15:49:18.811774 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:49:18.811757 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-s6jjq" Jul 07 15:49:18.812542 ip-10-0-132-65 kubenswrapper[2565]: E0707 15:49:18.812522 2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-2j52h\" is forbidden: User \"system:node:ip-10-0-132-65.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-65.ec2.internal' and this object" podUID="377033ac-01ab-4e31-8953-63b3ca057da4" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2j52h" Jul 07 15:49:18.819776 ip-10-0-132-65 kubenswrapper[2565]: E0707 15:49:18.819749 2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-s6jjq\" is forbidden: User \"system:node:ip-10-0-132-65.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-65.ec2.internal' and this object" podUID="accc5140-f794-42b1-a0a1-6141964184da" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-s6jjq" Jul 07 15:49:18.822199 ip-10-0-132-65 kubenswrapper[2565]: E0707 15:49:18.822178 2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-2j52h\" is forbidden: User \"system:node:ip-10-0-132-65.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-65.ec2.internal' and this object" podUID="377033ac-01ab-4e31-8953-63b3ca057da4" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2j52h" Jul 07 15:49:18.824647 ip-10-0-132-65 kubenswrapper[2565]: E0707 15:49:18.824626 2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-s6jjq\" is forbidden: User \"system:node:ip-10-0-132-65.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-65.ec2.internal' and this object" podUID="accc5140-f794-42b1-a0a1-6141964184da" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-s6jjq" Jul 07 15:49:18.862204 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:49:18.862170 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/377033ac-01ab-4e31-8953-63b3ca057da4-extensions-socket-volume\") pod \"377033ac-01ab-4e31-8953-63b3ca057da4\" (UID: \"377033ac-01ab-4e31-8953-63b3ca057da4\") " Jul 07 15:49:18.862353 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:49:18.862214 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vxr2f\" (UniqueName: \"kubernetes.io/projected/377033ac-01ab-4e31-8953-63b3ca057da4-kube-api-access-vxr2f\") pod \"377033ac-01ab-4e31-8953-63b3ca057da4\" (UID: \"377033ac-01ab-4e31-8953-63b3ca057da4\") " Jul 07 15:49:18.862637 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:49:18.862615 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/377033ac-01ab-4e31-8953-63b3ca057da4-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "377033ac-01ab-4e31-8953-63b3ca057da4" (UID: "377033ac-01ab-4e31-8953-63b3ca057da4"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 15:49:18.864240 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:49:18.864219 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/377033ac-01ab-4e31-8953-63b3ca057da4-kube-api-access-vxr2f" (OuterVolumeSpecName: "kube-api-access-vxr2f") pod "377033ac-01ab-4e31-8953-63b3ca057da4" (UID: "377033ac-01ab-4e31-8953-63b3ca057da4"). InnerVolumeSpecName "kube-api-access-vxr2f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 15:49:18.963155 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:49:18.963112 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kh6wb\" (UniqueName: \"kubernetes.io/projected/accc5140-f794-42b1-a0a1-6141964184da-kube-api-access-kh6wb\") pod \"accc5140-f794-42b1-a0a1-6141964184da\" (UID: \"accc5140-f794-42b1-a0a1-6141964184da\") " Jul 07 15:49:18.963324 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:49:18.963313 2565 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/377033ac-01ab-4e31-8953-63b3ca057da4-extensions-socket-volume\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 07 15:49:18.963367 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:49:18.963325 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vxr2f\" (UniqueName: \"kubernetes.io/projected/377033ac-01ab-4e31-8953-63b3ca057da4-kube-api-access-vxr2f\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 07 15:49:18.965134 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:49:18.965102 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/accc5140-f794-42b1-a0a1-6141964184da-kube-api-access-kh6wb" (OuterVolumeSpecName: "kube-api-access-kh6wb") pod "accc5140-f794-42b1-a0a1-6141964184da" (UID: "accc5140-f794-42b1-a0a1-6141964184da"). InnerVolumeSpecName "kube-api-access-kh6wb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 15:49:18.983310 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:49:18.983283 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wj6jx" Jul 07 15:49:18.990294 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:49:18.990278 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-nx6zw" Jul 07 15:49:19.064570 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:49:19.064476 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kh6wb\" (UniqueName: \"kubernetes.io/projected/accc5140-f794-42b1-a0a1-6141964184da-kube-api-access-kh6wb\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 07 15:49:19.133448 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:49:19.133423 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wj6jx"] Jul 07 15:49:19.135923 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:49:19.135893 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod395e1cf5_09da_4d48_aa8f_bc91863884ec.slice/crio-e2c2de3b49cbf89aa6476d1393a12063f100e88307c346dd7c1f0629af3e2745 WatchSource:0}: Error finding container e2c2de3b49cbf89aa6476d1393a12063f100e88307c346dd7c1f0629af3e2745: Status 404 returned error can't find the container with id e2c2de3b49cbf89aa6476d1393a12063f100e88307c346dd7c1f0629af3e2745 Jul 07 15:49:19.149779 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:49:19.149753 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-nx6zw"] Jul 07 15:49:19.155067 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:49:19.155044 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6555056e_fa42_4326_becf_9bd8eb0cfc21.slice/crio-8170a73c99107968fa49e20cd323f34199f45d13cce9c7837c152dac83386554 WatchSource:0}: Error finding container 8170a73c99107968fa49e20cd323f34199f45d13cce9c7837c152dac83386554: Status 404 returned error can't find the container with id 8170a73c99107968fa49e20cd323f34199f45d13cce9c7837c152dac83386554 Jul 07 15:49:19.661032 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:49:19.660991 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="377033ac-01ab-4e31-8953-63b3ca057da4" path="/var/lib/kubelet/pods/377033ac-01ab-4e31-8953-63b3ca057da4/volumes" Jul 07 15:49:19.661522 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:49:19.661485 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="accc5140-f794-42b1-a0a1-6141964184da" path="/var/lib/kubelet/pods/accc5140-f794-42b1-a0a1-6141964184da/volumes" Jul 07 15:49:19.750928 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:49:19.750896 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2j52h" Jul 07 15:49:19.750928 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:49:19.750916 2565 scope.go:117] "RemoveContainer" containerID="7d9d4e8c1fd73c44b20cf7d083b31af1f05665112c570a82e57af20b0e31f33f" Jul 07 15:49:19.752788 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:49:19.752754 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wj6jx" event={"ID":"395e1cf5-09da-4d48-aa8f-bc91863884ec","Type":"ContainerStarted","Data":"d9c4877105da5b0a9ab5e8c7a15e28b6105dc027fd813b2d4b7a688dde7a8eea"} Jul 07 15:49:19.752887 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:49:19.752814 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wj6jx" event={"ID":"395e1cf5-09da-4d48-aa8f-bc91863884ec","Type":"ContainerStarted","Data":"e2c2de3b49cbf89aa6476d1393a12063f100e88307c346dd7c1f0629af3e2745"} Jul 07 15:49:19.752887 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:49:19.752848 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wj6jx" Jul 07 15:49:19.754550 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:49:19.754528 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-nx6zw" event={"ID":"6555056e-fa42-4326-becf-9bd8eb0cfc21","Type":"ContainerStarted","Data":"3d5af5150a4da78dd33150dcf1565f4b6f752c9b67cdba7a797c032d53b5a5ba"} Jul 07 15:49:19.754646 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:49:19.754571 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-nx6zw" event={"ID":"6555056e-fa42-4326-becf-9bd8eb0cfc21","Type":"ContainerStarted","Data":"8170a73c99107968fa49e20cd323f34199f45d13cce9c7837c152dac83386554"} Jul 07 15:49:19.754646 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:49:19.754602 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-nx6zw" Jul 07 15:49:19.755841 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:49:19.755825 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-s6jjq" Jul 07 15:49:19.760744 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:49:19.760724 2565 scope.go:117] "RemoveContainer" containerID="cd07a1015b845d4fac450b29ee9babb09d8b21965318c798b54b6659816404df" Jul 07 15:49:19.777386 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:49:19.777345 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wj6jx" podStartSLOduration=1.777329691 podStartE2EDuration="1.777329691s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:49:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:49:19.775783125 +0000 UTC m=+498.732360592" watchObservedRunningTime="2026-07-07 15:49:19.777329691 +0000 UTC m=+498.733907157" Jul 07 15:49:19.797054 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:49:19.797009 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-nx6zw" podStartSLOduration=1.796994705 podStartE2EDuration="1.796994705s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:49:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:49:19.795827806 +0000 UTC m=+498.752405272" watchObservedRunningTime="2026-07-07 15:49:19.796994705 +0000 UTC m=+498.753572172" Jul 07 15:49:21.814983 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:49:21.814951 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:49:21.913689 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:49:21.913661 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:49:24.721575 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:49:24.721546 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-wvkvz" Jul 07 15:49:25.781889 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:49:25.781859 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:49:25.880270 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:49:25.880240 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:49:28.609556 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:49:28.609525 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:49:28.711634 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:49:28.711600 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:49:30.364976 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:49:30.364945 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:49:30.454979 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:49:30.454946 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:49:30.764703 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:49:30.764616 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wj6jx" Jul 07 15:49:30.764868 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:49:30.764818 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-nx6zw" Jul 07 15:49:30.842559 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:49:30.842529 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:49:30.937671 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:49:30.937631 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:49:32.348963 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:49:32.348928 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:49:32.481338 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:49:32.481308 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:49:35.182664 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:49:35.182615 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:49:35.282839 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:49:35.282801 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:49:42.883295 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:49:42.883265 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:49:42.963688 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:49:42.963653 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:49:47.577804 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:49:47.577768 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:49:47.677021 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:49:47.676988 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:49:48.058743 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:49:48.058637 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wj6jx"] Jul 07 15:49:48.058932 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:49:48.058908 2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wj6jx" podUID="395e1cf5-09da-4d48-aa8f-bc91863884ec" containerName="manager" containerID="cri-o://d9c4877105da5b0a9ab5e8c7a15e28b6105dc027fd813b2d4b7a688dde7a8eea" gracePeriod=10 Jul 07 15:49:48.305459 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:49:48.305431 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wj6jx" Jul 07 15:49:48.404750 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:49:48.404712 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8tbj5\" (UniqueName: \"kubernetes.io/projected/395e1cf5-09da-4d48-aa8f-bc91863884ec-kube-api-access-8tbj5\") pod \"395e1cf5-09da-4d48-aa8f-bc91863884ec\" (UID: \"395e1cf5-09da-4d48-aa8f-bc91863884ec\") " Jul 07 15:49:48.404925 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:49:48.404793 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/395e1cf5-09da-4d48-aa8f-bc91863884ec-extensions-socket-volume\") pod \"395e1cf5-09da-4d48-aa8f-bc91863884ec\" (UID: \"395e1cf5-09da-4d48-aa8f-bc91863884ec\") " Jul 07 15:49:48.405183 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:49:48.405160 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/395e1cf5-09da-4d48-aa8f-bc91863884ec-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "395e1cf5-09da-4d48-aa8f-bc91863884ec" (UID: "395e1cf5-09da-4d48-aa8f-bc91863884ec"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 15:49:48.406840 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:49:48.406818 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/395e1cf5-09da-4d48-aa8f-bc91863884ec-kube-api-access-8tbj5" (OuterVolumeSpecName: "kube-api-access-8tbj5") pod "395e1cf5-09da-4d48-aa8f-bc91863884ec" (UID: "395e1cf5-09da-4d48-aa8f-bc91863884ec"). InnerVolumeSpecName "kube-api-access-8tbj5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 15:49:48.505856 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:49:48.505824 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8tbj5\" (UniqueName: \"kubernetes.io/projected/395e1cf5-09da-4d48-aa8f-bc91863884ec-kube-api-access-8tbj5\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 07 15:49:48.505856 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:49:48.505850 2565 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/395e1cf5-09da-4d48-aa8f-bc91863884ec-extensions-socket-volume\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 07 15:49:48.888998 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:49:48.888958 2565 generic.go:356] "Generic (PLEG): container finished" podID="395e1cf5-09da-4d48-aa8f-bc91863884ec" containerID="d9c4877105da5b0a9ab5e8c7a15e28b6105dc027fd813b2d4b7a688dde7a8eea" exitCode=0 Jul 07 15:49:48.889409 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:49:48.889010 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wj6jx" Jul 07 15:49:48.889409 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:49:48.889046 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wj6jx" event={"ID":"395e1cf5-09da-4d48-aa8f-bc91863884ec","Type":"ContainerDied","Data":"d9c4877105da5b0a9ab5e8c7a15e28b6105dc027fd813b2d4b7a688dde7a8eea"} Jul 07 15:49:48.889409 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:49:48.889107 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wj6jx" event={"ID":"395e1cf5-09da-4d48-aa8f-bc91863884ec","Type":"ContainerDied","Data":"e2c2de3b49cbf89aa6476d1393a12063f100e88307c346dd7c1f0629af3e2745"} Jul 07 15:49:48.889409 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:49:48.889130 2565 scope.go:117] "RemoveContainer" containerID="d9c4877105da5b0a9ab5e8c7a15e28b6105dc027fd813b2d4b7a688dde7a8eea" Jul 07 15:49:48.898139 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:49:48.898122 2565 scope.go:117] "RemoveContainer" containerID="d9c4877105da5b0a9ab5e8c7a15e28b6105dc027fd813b2d4b7a688dde7a8eea" Jul 07 15:49:48.898401 ip-10-0-132-65 kubenswrapper[2565]: E0707 15:49:48.898383 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d9c4877105da5b0a9ab5e8c7a15e28b6105dc027fd813b2d4b7a688dde7a8eea\": container with ID starting with d9c4877105da5b0a9ab5e8c7a15e28b6105dc027fd813b2d4b7a688dde7a8eea not found: ID does not exist" containerID="d9c4877105da5b0a9ab5e8c7a15e28b6105dc027fd813b2d4b7a688dde7a8eea" Jul 07 15:49:48.898447 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:49:48.898410 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9c4877105da5b0a9ab5e8c7a15e28b6105dc027fd813b2d4b7a688dde7a8eea"} err="failed to get container status \"d9c4877105da5b0a9ab5e8c7a15e28b6105dc027fd813b2d4b7a688dde7a8eea\": rpc error: code = NotFound desc = could not find container \"d9c4877105da5b0a9ab5e8c7a15e28b6105dc027fd813b2d4b7a688dde7a8eea\": container with ID starting with d9c4877105da5b0a9ab5e8c7a15e28b6105dc027fd813b2d4b7a688dde7a8eea not found: ID does not exist" Jul 07 15:49:48.920177 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:49:48.920147 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wj6jx"] Jul 07 15:49:48.927595 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:49:48.927567 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wj6jx"] Jul 07 15:49:49.660600 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:49:49.660563 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="395e1cf5-09da-4d48-aa8f-bc91863884ec" path="/var/lib/kubelet/pods/395e1cf5-09da-4d48-aa8f-bc91863884ec/volumes" Jul 07 15:49:52.390476 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:49:52.390442 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:49:52.494385 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:49:52.494344 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:49:56.410243 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:49:56.410167 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:49:56.508109 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:49:56.508078 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:50:04.195938 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:04.195890 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:50:04.296883 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:04.296848 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:50:04.569025 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:04.568992 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-tqwxj"] Jul 07 15:50:04.569343 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:04.569330 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="395e1cf5-09da-4d48-aa8f-bc91863884ec" containerName="manager" Jul 07 15:50:04.569389 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:04.569344 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="395e1cf5-09da-4d48-aa8f-bc91863884ec" containerName="manager" Jul 07 15:50:04.569422 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:04.569401 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="395e1cf5-09da-4d48-aa8f-bc91863884ec" containerName="manager" Jul 07 15:50:04.572240 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:04.572222 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-tqwxj" Jul 07 15:50:04.575592 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:04.575574 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-kqfcz\"" Jul 07 15:50:04.621474 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:04.621444 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-tqwxj"] Jul 07 15:50:04.638771 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:04.638740 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:50:04.742663 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:04.742632 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:50:04.745712 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:04.745690 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8b5b\" (UniqueName: \"kubernetes.io/projected/ea743bfe-7a54-46a1-8f15-cb6e93ee0f98-kube-api-access-k8b5b\") pod \"maas-default-gateway-openshift-default-5896d4dd49-tqwxj\" (UID: \"ea743bfe-7a54-46a1-8f15-cb6e93ee0f98\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-tqwxj" Jul 07 15:50:04.745822 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:04.745722 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea743bfe-7a54-46a1-8f15-cb6e93ee0f98-workload-certs\") pod \"maas-default-gateway-openshift-default-5896d4dd49-tqwxj\" (UID: \"ea743bfe-7a54-46a1-8f15-cb6e93ee0f98\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-tqwxj" Jul 07 15:50:04.745822 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:04.745744 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea743bfe-7a54-46a1-8f15-cb6e93ee0f98-istio-data\") pod \"maas-default-gateway-openshift-default-5896d4dd49-tqwxj\" (UID: \"ea743bfe-7a54-46a1-8f15-cb6e93ee0f98\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-tqwxj" Jul 07 15:50:04.745822 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:04.745779 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea743bfe-7a54-46a1-8f15-cb6e93ee0f98-istio-envoy\") pod \"maas-default-gateway-openshift-default-5896d4dd49-tqwxj\" (UID: \"ea743bfe-7a54-46a1-8f15-cb6e93ee0f98\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-tqwxj" Jul 07 15:50:04.745822 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:04.745809 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea743bfe-7a54-46a1-8f15-cb6e93ee0f98-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5896d4dd49-tqwxj\" (UID: \"ea743bfe-7a54-46a1-8f15-cb6e93ee0f98\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-tqwxj" Jul 07 15:50:04.745954 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:04.745862 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea743bfe-7a54-46a1-8f15-cb6e93ee0f98-credential-socket\") pod \"maas-default-gateway-openshift-default-5896d4dd49-tqwxj\" (UID: \"ea743bfe-7a54-46a1-8f15-cb6e93ee0f98\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-tqwxj" Jul 07 15:50:04.745954 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:04.745888 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea743bfe-7a54-46a1-8f15-cb6e93ee0f98-istio-token\") pod \"maas-default-gateway-openshift-default-5896d4dd49-tqwxj\" (UID: \"ea743bfe-7a54-46a1-8f15-cb6e93ee0f98\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-tqwxj" Jul 07 15:50:04.745954 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:04.745914 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea743bfe-7a54-46a1-8f15-cb6e93ee0f98-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5896d4dd49-tqwxj\" (UID: \"ea743bfe-7a54-46a1-8f15-cb6e93ee0f98\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-tqwxj" Jul 07 15:50:04.746048 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:04.745982 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea743bfe-7a54-46a1-8f15-cb6e93ee0f98-workload-socket\") pod \"maas-default-gateway-openshift-default-5896d4dd49-tqwxj\" (UID: \"ea743bfe-7a54-46a1-8f15-cb6e93ee0f98\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-tqwxj" Jul 07 15:50:04.847446 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:04.847358 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8b5b\" (UniqueName: \"kubernetes.io/projected/ea743bfe-7a54-46a1-8f15-cb6e93ee0f98-kube-api-access-k8b5b\") pod \"maas-default-gateway-openshift-default-5896d4dd49-tqwxj\" (UID: \"ea743bfe-7a54-46a1-8f15-cb6e93ee0f98\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-tqwxj" Jul 07 15:50:04.847446 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:04.847399 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea743bfe-7a54-46a1-8f15-cb6e93ee0f98-workload-certs\") pod \"maas-default-gateway-openshift-default-5896d4dd49-tqwxj\" (UID: \"ea743bfe-7a54-46a1-8f15-cb6e93ee0f98\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-tqwxj" Jul 07 15:50:04.847446 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:04.847415 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea743bfe-7a54-46a1-8f15-cb6e93ee0f98-istio-data\") pod \"maas-default-gateway-openshift-default-5896d4dd49-tqwxj\" (UID: \"ea743bfe-7a54-46a1-8f15-cb6e93ee0f98\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-tqwxj" Jul 07 15:50:04.847446 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:04.847440 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea743bfe-7a54-46a1-8f15-cb6e93ee0f98-istio-envoy\") pod \"maas-default-gateway-openshift-default-5896d4dd49-tqwxj\" (UID: \"ea743bfe-7a54-46a1-8f15-cb6e93ee0f98\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-tqwxj" Jul 07 15:50:04.847781 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:04.847610 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea743bfe-7a54-46a1-8f15-cb6e93ee0f98-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5896d4dd49-tqwxj\" (UID: \"ea743bfe-7a54-46a1-8f15-cb6e93ee0f98\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-tqwxj" Jul 07 15:50:04.847781 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:04.847684 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea743bfe-7a54-46a1-8f15-cb6e93ee0f98-credential-socket\") pod \"maas-default-gateway-openshift-default-5896d4dd49-tqwxj\" (UID: \"ea743bfe-7a54-46a1-8f15-cb6e93ee0f98\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-tqwxj" Jul 07 15:50:04.847781 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:04.847722 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea743bfe-7a54-46a1-8f15-cb6e93ee0f98-istio-token\") pod \"maas-default-gateway-openshift-default-5896d4dd49-tqwxj\" (UID: \"ea743bfe-7a54-46a1-8f15-cb6e93ee0f98\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-tqwxj" Jul 07 15:50:04.847781 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:04.847765 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea743bfe-7a54-46a1-8f15-cb6e93ee0f98-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5896d4dd49-tqwxj\" (UID: \"ea743bfe-7a54-46a1-8f15-cb6e93ee0f98\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-tqwxj" Jul 07 15:50:04.847989 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:04.847850 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea743bfe-7a54-46a1-8f15-cb6e93ee0f98-workload-socket\") pod \"maas-default-gateway-openshift-default-5896d4dd49-tqwxj\" (UID: \"ea743bfe-7a54-46a1-8f15-cb6e93ee0f98\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-tqwxj" Jul 07 15:50:04.847989 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:04.847881 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea743bfe-7a54-46a1-8f15-cb6e93ee0f98-istio-data\") pod \"maas-default-gateway-openshift-default-5896d4dd49-tqwxj\" (UID: \"ea743bfe-7a54-46a1-8f15-cb6e93ee0f98\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-tqwxj" Jul 07 15:50:04.847989 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:04.847969 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea743bfe-7a54-46a1-8f15-cb6e93ee0f98-workload-certs\") pod \"maas-default-gateway-openshift-default-5896d4dd49-tqwxj\" (UID: \"ea743bfe-7a54-46a1-8f15-cb6e93ee0f98\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-tqwxj" Jul 07 15:50:04.848167 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:04.848143 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea743bfe-7a54-46a1-8f15-cb6e93ee0f98-credential-socket\") pod \"maas-default-gateway-openshift-default-5896d4dd49-tqwxj\" (UID: \"ea743bfe-7a54-46a1-8f15-cb6e93ee0f98\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-tqwxj" Jul 07 15:50:04.848240 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:04.848168 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea743bfe-7a54-46a1-8f15-cb6e93ee0f98-workload-socket\") pod \"maas-default-gateway-openshift-default-5896d4dd49-tqwxj\" (UID: \"ea743bfe-7a54-46a1-8f15-cb6e93ee0f98\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-tqwxj" Jul 07 15:50:04.848372 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:04.848349 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea743bfe-7a54-46a1-8f15-cb6e93ee0f98-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5896d4dd49-tqwxj\" (UID: \"ea743bfe-7a54-46a1-8f15-cb6e93ee0f98\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-tqwxj" Jul 07 15:50:04.849763 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:04.849745 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea743bfe-7a54-46a1-8f15-cb6e93ee0f98-istio-envoy\") pod \"maas-default-gateway-openshift-default-5896d4dd49-tqwxj\" (UID: \"ea743bfe-7a54-46a1-8f15-cb6e93ee0f98\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-tqwxj" Jul 07 15:50:04.850103 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:04.850087 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea743bfe-7a54-46a1-8f15-cb6e93ee0f98-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5896d4dd49-tqwxj\" (UID: \"ea743bfe-7a54-46a1-8f15-cb6e93ee0f98\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-tqwxj" Jul 07 15:50:04.859216 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:04.859193 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea743bfe-7a54-46a1-8f15-cb6e93ee0f98-istio-token\") pod \"maas-default-gateway-openshift-default-5896d4dd49-tqwxj\" (UID: \"ea743bfe-7a54-46a1-8f15-cb6e93ee0f98\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-tqwxj" Jul 07 15:50:04.861172 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:04.861138 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8b5b\" (UniqueName: \"kubernetes.io/projected/ea743bfe-7a54-46a1-8f15-cb6e93ee0f98-kube-api-access-k8b5b\") pod \"maas-default-gateway-openshift-default-5896d4dd49-tqwxj\" (UID: \"ea743bfe-7a54-46a1-8f15-cb6e93ee0f98\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-tqwxj" Jul 07 15:50:04.881996 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:04.881958 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-tqwxj" Jul 07 15:50:05.043272 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:05.043248 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-tqwxj"] Jul 07 15:50:05.045076 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:50:05.045051 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea743bfe_7a54_46a1_8f15_cb6e93ee0f98.slice/crio-a167c94399b44524991d632831b2b11dea53cbfccd6ab925ab459d6a71415d39 WatchSource:0}: Error finding container a167c94399b44524991d632831b2b11dea53cbfccd6ab925ab459d6a71415d39: Status 404 returned error can't find the container with id a167c94399b44524991d632831b2b11dea53cbfccd6ab925ab459d6a71415d39 Jul 07 15:50:05.047099 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:05.047069 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 07 15:50:05.047181 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:05.047134 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 07 15:50:05.047181 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:05.047160 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 07 15:50:05.968861 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:05.968821 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-tqwxj" event={"ID":"ea743bfe-7a54-46a1-8f15-cb6e93ee0f98","Type":"ContainerStarted","Data":"1e42d0d45f890b584a3b5682c03d9b7310633f85492a5389f2da5c2b33334539"} Jul 07 15:50:05.969311 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:05.968886 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-tqwxj" event={"ID":"ea743bfe-7a54-46a1-8f15-cb6e93ee0f98","Type":"ContainerStarted","Data":"a167c94399b44524991d632831b2b11dea53cbfccd6ab925ab459d6a71415d39"} Jul 07 15:50:06.027849 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:06.027795 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-tqwxj" podStartSLOduration=2.027777047 podStartE2EDuration="2.027777047s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:50:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:50:06.027142789 +0000 UTC m=+544.983720256" watchObservedRunningTime="2026-07-07 15:50:06.027777047 +0000 UTC m=+544.984354513" Jul 07 15:50:06.883089 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:06.883042 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-tqwxj" Jul 07 15:50:06.887794 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:06.887768 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-tqwxj" Jul 07 15:50:06.972891 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:06.972855 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-tqwxj" Jul 07 15:50:06.973934 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:06.973908 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-tqwxj" Jul 07 15:50:07.560234 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:07.560203 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:50:07.597004 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:07.596967 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-h8qwd"] Jul 07 15:50:07.600066 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:07.600043 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-h8qwd" Jul 07 15:50:07.603481 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:07.603443 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-jglv8\"" Jul 07 15:50:07.603738 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:07.603552 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 07 15:50:07.615051 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:07.615023 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-h8qwd"] Jul 07 15:50:07.664216 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:07.664161 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:50:07.676116 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:07.676090 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-h8qwd"] Jul 07 15:50:07.766154 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:07.766116 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:50:07.772375 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:07.772351 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/fff457fd-d9ad-448e-a5ab-1333ff3307ec-config-file\") pod \"limitador-limitador-64c94fd64f-h8qwd\" (UID: \"fff457fd-d9ad-448e-a5ab-1333ff3307ec\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-h8qwd" Jul 07 15:50:07.772567 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:07.772388 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dswpk\" (UniqueName: \"kubernetes.io/projected/fff457fd-d9ad-448e-a5ab-1333ff3307ec-kube-api-access-dswpk\") pod \"limitador-limitador-64c94fd64f-h8qwd\" (UID: \"fff457fd-d9ad-448e-a5ab-1333ff3307ec\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-h8qwd" Jul 07 15:50:07.874028 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:07.873938 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/fff457fd-d9ad-448e-a5ab-1333ff3307ec-config-file\") pod \"limitador-limitador-64c94fd64f-h8qwd\" (UID: \"fff457fd-d9ad-448e-a5ab-1333ff3307ec\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-h8qwd" Jul 07 15:50:07.874028 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:07.873987 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dswpk\" (UniqueName: \"kubernetes.io/projected/fff457fd-d9ad-448e-a5ab-1333ff3307ec-kube-api-access-dswpk\") pod \"limitador-limitador-64c94fd64f-h8qwd\" (UID: \"fff457fd-d9ad-448e-a5ab-1333ff3307ec\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-h8qwd" Jul 07 15:50:07.874586 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:07.874567 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/fff457fd-d9ad-448e-a5ab-1333ff3307ec-config-file\") pod \"limitador-limitador-64c94fd64f-h8qwd\" (UID: \"fff457fd-d9ad-448e-a5ab-1333ff3307ec\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-h8qwd" Jul 07 15:50:07.888357 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:07.888326 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dswpk\" (UniqueName: \"kubernetes.io/projected/fff457fd-d9ad-448e-a5ab-1333ff3307ec-kube-api-access-dswpk\") pod \"limitador-limitador-64c94fd64f-h8qwd\" (UID: \"fff457fd-d9ad-448e-a5ab-1333ff3307ec\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-h8qwd" Jul 07 15:50:07.912935 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:07.912909 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-h8qwd" Jul 07 15:50:08.068890 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:08.068793 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-h8qwd"] Jul 07 15:50:08.071192 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:50:08.071165 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfff457fd_d9ad_448e_a5ab_1333ff3307ec.slice/crio-c2eb5231b8bcf7a1ce9c965a9e57affbfc6533756a862efb678b5ad547d5fc9f WatchSource:0}: Error finding container c2eb5231b8bcf7a1ce9c965a9e57affbfc6533756a862efb678b5ad547d5fc9f: Status 404 returned error can't find the container with id c2eb5231b8bcf7a1ce9c965a9e57affbfc6533756a862efb678b5ad547d5fc9f Jul 07 15:50:08.086577 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:08.086552 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:50:08.183111 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:08.183024 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:50:08.762996 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:08.762965 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-58cd955675-rgk2k"] Jul 07 15:50:08.765814 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:08.765793 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-rgk2k" Jul 07 15:50:08.781662 ip-10-0-132-65 kubenswrapper[2565]: E0707 15:50:08.781630 2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"authorino-58cd955675-rgk2k\" is forbidden: User \"system:node:ip-10-0-132-65.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-65.ec2.internal' and this object" podUID="5ae6fbf7-3ed8-4c1a-aacd-ea5331026f70" pod="kuadrant-system/authorino-58cd955675-rgk2k" Jul 07 15:50:08.784261 ip-10-0-132-65 kubenswrapper[2565]: E0707 15:50:08.784238 2565 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"authorino-authorino-dockercfg-ctdxm\" is forbidden: User \"system:node:ip-10-0-132-65.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-65.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-ctdxm\"" type="*v1.Secret" Jul 07 15:50:08.810884 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:08.810857 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:50:08.811048 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:08.811010 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-rgk2k"] Jul 07 15:50:08.882274 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:08.882233 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g762d\" (UniqueName: \"kubernetes.io/projected/5ae6fbf7-3ed8-4c1a-aacd-ea5331026f70-kube-api-access-g762d\") pod \"authorino-58cd955675-rgk2k\" (UID: \"5ae6fbf7-3ed8-4c1a-aacd-ea5331026f70\") " pod="kuadrant-system/authorino-58cd955675-rgk2k" Jul 07 15:50:08.901972 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:08.901942 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:50:08.982853 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:08.982814 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g762d\" (UniqueName: \"kubernetes.io/projected/5ae6fbf7-3ed8-4c1a-aacd-ea5331026f70-kube-api-access-g762d\") pod \"authorino-58cd955675-rgk2k\" (UID: \"5ae6fbf7-3ed8-4c1a-aacd-ea5331026f70\") " pod="kuadrant-system/authorino-58cd955675-rgk2k" Jul 07 15:50:08.988344 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:08.988288 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-h8qwd" event={"ID":"fff457fd-d9ad-448e-a5ab-1333ff3307ec","Type":"ContainerStarted","Data":"c2eb5231b8bcf7a1ce9c965a9e57affbfc6533756a862efb678b5ad547d5fc9f"} Jul 07 15:50:08.994407 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:08.994380 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g762d\" (UniqueName: \"kubernetes.io/projected/5ae6fbf7-3ed8-4c1a-aacd-ea5331026f70-kube-api-access-g762d\") pod \"authorino-58cd955675-rgk2k\" (UID: \"5ae6fbf7-3ed8-4c1a-aacd-ea5331026f70\") " pod="kuadrant-system/authorino-58cd955675-rgk2k" Jul 07 15:50:09.743937 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:09.743905 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-ctdxm\"" Jul 07 15:50:09.745991 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:09.745970 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-rgk2k" Jul 07 15:50:10.185784 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:10.185743 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:50:11.063825 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:11.063803 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-rgk2k"] Jul 07 15:50:11.065586 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:50:11.065560 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5ae6fbf7_3ed8_4c1a_aacd_ea5331026f70.slice/crio-1f1de248d4b4dee582fa02d82bd893767c9bd35ed6b64a34a1c55f5fef5194df WatchSource:0}: Error finding container 1f1de248d4b4dee582fa02d82bd893767c9bd35ed6b64a34a1c55f5fef5194df: Status 404 returned error can't find the container with id 1f1de248d4b4dee582fa02d82bd893767c9bd35ed6b64a34a1c55f5fef5194df Jul 07 15:50:12.013724 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:12.013682 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-h8qwd" event={"ID":"fff457fd-d9ad-448e-a5ab-1333ff3307ec","Type":"ContainerStarted","Data":"447bf2a6b801098f6bff71bd9004e2bdfec80ccf207e834c85adbc4fea04f39b"} Jul 07 15:50:12.014092 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:12.014071 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-h8qwd" Jul 07 15:50:12.015378 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:12.015348 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-rgk2k" event={"ID":"5ae6fbf7-3ed8-4c1a-aacd-ea5331026f70","Type":"ContainerStarted","Data":"1f1de248d4b4dee582fa02d82bd893767c9bd35ed6b64a34a1c55f5fef5194df"} Jul 07 15:50:12.054329 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:12.054267 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-64c94fd64f-h8qwd" podStartSLOduration=2.17702743 podStartE2EDuration="5.054243861s" totalImagesPullingTime="2.877216431s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:50:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:50:12.05127681 +0000 UTC m=+551.007854278" watchObservedRunningTime="2026-07-07 15:50:12.054243861 +0000 UTC m=+551.010821330" Jul 07 15:50:12.818379 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:12.818329 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:50:12.918861 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:12.918748 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:50:13.260356 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:13.260271 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-rgk2k"] Jul 07 15:50:14.026365 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:14.026326 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-rgk2k" event={"ID":"5ae6fbf7-3ed8-4c1a-aacd-ea5331026f70","Type":"ContainerStarted","Data":"b27b4c1acf846c72196595c87d8dea79be30a3112dfafe830d789c3585120f55"} Jul 07 15:50:14.026916 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:14.026422 2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/authorino-58cd955675-rgk2k" podUID="5ae6fbf7-3ed8-4c1a-aacd-ea5331026f70" containerName="authorino" containerID="cri-o://b27b4c1acf846c72196595c87d8dea79be30a3112dfafe830d789c3585120f55" gracePeriod=30 Jul 07 15:50:14.051006 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:14.050960 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-58cd955675-rgk2k" podStartSLOduration=3.233555498 podStartE2EDuration="6.050944718s" totalImagesPullingTime="2.81738922s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:50:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:50:14.048176463 +0000 UTC m=+553.004753930" watchObservedRunningTime="2026-07-07 15:50:14.050944718 +0000 UTC m=+553.007522184" Jul 07 15:50:14.282575 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:14.282546 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-rgk2k" Jul 07 15:50:14.432372 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:14.432289 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g762d\" (UniqueName: \"kubernetes.io/projected/5ae6fbf7-3ed8-4c1a-aacd-ea5331026f70-kube-api-access-g762d\") pod \"5ae6fbf7-3ed8-4c1a-aacd-ea5331026f70\" (UID: \"5ae6fbf7-3ed8-4c1a-aacd-ea5331026f70\") " Jul 07 15:50:14.434423 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:14.434391 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ae6fbf7-3ed8-4c1a-aacd-ea5331026f70-kube-api-access-g762d" (OuterVolumeSpecName: "kube-api-access-g762d") pod "5ae6fbf7-3ed8-4c1a-aacd-ea5331026f70" (UID: "5ae6fbf7-3ed8-4c1a-aacd-ea5331026f70"). InnerVolumeSpecName "kube-api-access-g762d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 15:50:14.533041 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:14.533010 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g762d\" (UniqueName: \"kubernetes.io/projected/5ae6fbf7-3ed8-4c1a-aacd-ea5331026f70-kube-api-access-g762d\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 07 15:50:15.031884 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:15.031855 2565 generic.go:356] "Generic (PLEG): container finished" podID="5ae6fbf7-3ed8-4c1a-aacd-ea5331026f70" containerID="b27b4c1acf846c72196595c87d8dea79be30a3112dfafe830d789c3585120f55" exitCode=0 Jul 07 15:50:15.032278 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:15.031893 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-rgk2k" Jul 07 15:50:15.032278 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:15.031933 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-rgk2k" event={"ID":"5ae6fbf7-3ed8-4c1a-aacd-ea5331026f70","Type":"ContainerDied","Data":"b27b4c1acf846c72196595c87d8dea79be30a3112dfafe830d789c3585120f55"} Jul 07 15:50:15.032278 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:15.031962 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-rgk2k" event={"ID":"5ae6fbf7-3ed8-4c1a-aacd-ea5331026f70","Type":"ContainerDied","Data":"1f1de248d4b4dee582fa02d82bd893767c9bd35ed6b64a34a1c55f5fef5194df"} Jul 07 15:50:15.032278 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:15.031980 2565 scope.go:117] "RemoveContainer" containerID="b27b4c1acf846c72196595c87d8dea79be30a3112dfafe830d789c3585120f55" Jul 07 15:50:15.040862 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:15.040847 2565 scope.go:117] "RemoveContainer" containerID="b27b4c1acf846c72196595c87d8dea79be30a3112dfafe830d789c3585120f55" Jul 07 15:50:15.041105 ip-10-0-132-65 kubenswrapper[2565]: E0707 15:50:15.041087 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b27b4c1acf846c72196595c87d8dea79be30a3112dfafe830d789c3585120f55\": container with ID starting with b27b4c1acf846c72196595c87d8dea79be30a3112dfafe830d789c3585120f55 not found: ID does not exist" containerID="b27b4c1acf846c72196595c87d8dea79be30a3112dfafe830d789c3585120f55" Jul 07 15:50:15.041156 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:15.041115 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b27b4c1acf846c72196595c87d8dea79be30a3112dfafe830d789c3585120f55"} err="failed to get container status \"b27b4c1acf846c72196595c87d8dea79be30a3112dfafe830d789c3585120f55\": rpc error: code = NotFound desc = could not find container \"b27b4c1acf846c72196595c87d8dea79be30a3112dfafe830d789c3585120f55\": container with ID starting with b27b4c1acf846c72196595c87d8dea79be30a3112dfafe830d789c3585120f55 not found: ID does not exist" Jul 07 15:50:15.079303 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:15.079269 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-rgk2k"] Jul 07 15:50:15.094258 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:15.094230 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-58cd955675-rgk2k"] Jul 07 15:50:15.659944 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:15.659911 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5ae6fbf7-3ed8-4c1a-aacd-ea5331026f70" path="/var/lib/kubelet/pods/5ae6fbf7-3ed8-4c1a-aacd-ea5331026f70/volumes" Jul 07 15:50:22.966268 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:22.966230 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-h8qwd"] Jul 07 15:50:22.966712 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:22.966443 2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-64c94fd64f-h8qwd" podUID="fff457fd-d9ad-448e-a5ab-1333ff3307ec" containerName="limitador" containerID="cri-o://447bf2a6b801098f6bff71bd9004e2bdfec80ccf207e834c85adbc4fea04f39b" gracePeriod=30 Jul 07 15:50:22.967226 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:22.967206 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-h8qwd" Jul 07 15:50:23.521886 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:23.521857 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-h8qwd" Jul 07 15:50:23.614345 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:23.614306 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/fff457fd-d9ad-448e-a5ab-1333ff3307ec-config-file\") pod \"fff457fd-d9ad-448e-a5ab-1333ff3307ec\" (UID: \"fff457fd-d9ad-448e-a5ab-1333ff3307ec\") " Jul 07 15:50:23.614539 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:23.614374 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dswpk\" (UniqueName: \"kubernetes.io/projected/fff457fd-d9ad-448e-a5ab-1333ff3307ec-kube-api-access-dswpk\") pod \"fff457fd-d9ad-448e-a5ab-1333ff3307ec\" (UID: \"fff457fd-d9ad-448e-a5ab-1333ff3307ec\") " Jul 07 15:50:23.614713 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:23.614690 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fff457fd-d9ad-448e-a5ab-1333ff3307ec-config-file" (OuterVolumeSpecName: "config-file") pod "fff457fd-d9ad-448e-a5ab-1333ff3307ec" (UID: "fff457fd-d9ad-448e-a5ab-1333ff3307ec"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 15:50:23.616386 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:23.616362 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fff457fd-d9ad-448e-a5ab-1333ff3307ec-kube-api-access-dswpk" (OuterVolumeSpecName: "kube-api-access-dswpk") pod "fff457fd-d9ad-448e-a5ab-1333ff3307ec" (UID: "fff457fd-d9ad-448e-a5ab-1333ff3307ec"). InnerVolumeSpecName "kube-api-access-dswpk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 15:50:23.715433 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:23.715403 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dswpk\" (UniqueName: \"kubernetes.io/projected/fff457fd-d9ad-448e-a5ab-1333ff3307ec-kube-api-access-dswpk\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 07 15:50:23.715433 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:23.715427 2565 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/fff457fd-d9ad-448e-a5ab-1333ff3307ec-config-file\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 07 15:50:24.075025 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:24.074994 2565 generic.go:356] "Generic (PLEG): container finished" podID="fff457fd-d9ad-448e-a5ab-1333ff3307ec" containerID="447bf2a6b801098f6bff71bd9004e2bdfec80ccf207e834c85adbc4fea04f39b" exitCode=0 Jul 07 15:50:24.075567 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:24.075038 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-h8qwd" Jul 07 15:50:24.075567 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:24.075075 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-h8qwd" event={"ID":"fff457fd-d9ad-448e-a5ab-1333ff3307ec","Type":"ContainerDied","Data":"447bf2a6b801098f6bff71bd9004e2bdfec80ccf207e834c85adbc4fea04f39b"} Jul 07 15:50:24.075567 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:24.075129 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-h8qwd" event={"ID":"fff457fd-d9ad-448e-a5ab-1333ff3307ec","Type":"ContainerDied","Data":"c2eb5231b8bcf7a1ce9c965a9e57affbfc6533756a862efb678b5ad547d5fc9f"} Jul 07 15:50:24.075567 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:24.075165 2565 scope.go:117] "RemoveContainer" containerID="447bf2a6b801098f6bff71bd9004e2bdfec80ccf207e834c85adbc4fea04f39b" Jul 07 15:50:24.083741 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:24.083722 2565 scope.go:117] "RemoveContainer" containerID="447bf2a6b801098f6bff71bd9004e2bdfec80ccf207e834c85adbc4fea04f39b" Jul 07 15:50:24.083994 ip-10-0-132-65 kubenswrapper[2565]: E0707 15:50:24.083975 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"447bf2a6b801098f6bff71bd9004e2bdfec80ccf207e834c85adbc4fea04f39b\": container with ID starting with 447bf2a6b801098f6bff71bd9004e2bdfec80ccf207e834c85adbc4fea04f39b not found: ID does not exist" containerID="447bf2a6b801098f6bff71bd9004e2bdfec80ccf207e834c85adbc4fea04f39b" Jul 07 15:50:24.084038 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:24.084003 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"447bf2a6b801098f6bff71bd9004e2bdfec80ccf207e834c85adbc4fea04f39b"} err="failed to get container status \"447bf2a6b801098f6bff71bd9004e2bdfec80ccf207e834c85adbc4fea04f39b\": rpc error: code = NotFound desc = could not find container \"447bf2a6b801098f6bff71bd9004e2bdfec80ccf207e834c85adbc4fea04f39b\": container with ID starting with 447bf2a6b801098f6bff71bd9004e2bdfec80ccf207e834c85adbc4fea04f39b not found: ID does not exist" Jul 07 15:50:24.098341 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:24.098318 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-h8qwd"] Jul 07 15:50:24.102923 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:24.102902 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-h8qwd"] Jul 07 15:50:25.660287 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:25.660252 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fff457fd-d9ad-448e-a5ab-1333ff3307ec" path="/var/lib/kubelet/pods/fff457fd-d9ad-448e-a5ab-1333ff3307ec/volumes" Jul 07 15:50:28.713741 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:28.713711 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/postgres-69fb4594cf-wrv28"] Jul 07 15:50:28.714126 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:28.714058 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ae6fbf7-3ed8-4c1a-aacd-ea5331026f70" containerName="authorino" Jul 07 15:50:28.714126 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:28.714069 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ae6fbf7-3ed8-4c1a-aacd-ea5331026f70" containerName="authorino" Jul 07 15:50:28.714126 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:28.714078 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fff457fd-d9ad-448e-a5ab-1333ff3307ec" containerName="limitador" Jul 07 15:50:28.714126 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:28.714084 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="fff457fd-d9ad-448e-a5ab-1333ff3307ec" containerName="limitador" Jul 07 15:50:28.714282 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:28.714147 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="fff457fd-d9ad-448e-a5ab-1333ff3307ec" containerName="limitador" Jul 07 15:50:28.714282 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:28.714159 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="5ae6fbf7-3ed8-4c1a-aacd-ea5331026f70" containerName="authorino" Jul 07 15:50:28.717107 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:28.717090 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/postgres-69fb4594cf-wrv28" Jul 07 15:50:28.720370 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:28.720350 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"postgres-creds\"" Jul 07 15:50:28.720478 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:28.720397 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-hw6dg\"" Jul 07 15:50:28.732532 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:28.732481 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/postgres-69fb4594cf-wrv28"] Jul 07 15:50:28.756873 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:28.756847 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/3fdac274-a101-4e09-9e15-83842c752d99-data\") pod \"postgres-69fb4594cf-wrv28\" (UID: \"3fdac274-a101-4e09-9e15-83842c752d99\") " pod="opendatahub/postgres-69fb4594cf-wrv28" Jul 07 15:50:28.757019 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:28.756991 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9j6gd\" (UniqueName: \"kubernetes.io/projected/3fdac274-a101-4e09-9e15-83842c752d99-kube-api-access-9j6gd\") pod \"postgres-69fb4594cf-wrv28\" (UID: \"3fdac274-a101-4e09-9e15-83842c752d99\") " pod="opendatahub/postgres-69fb4594cf-wrv28" Jul 07 15:50:28.763333 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:28.763307 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:50:28.858349 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:28.858313 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9j6gd\" (UniqueName: \"kubernetes.io/projected/3fdac274-a101-4e09-9e15-83842c752d99-kube-api-access-9j6gd\") pod \"postgres-69fb4594cf-wrv28\" (UID: \"3fdac274-a101-4e09-9e15-83842c752d99\") " pod="opendatahub/postgres-69fb4594cf-wrv28" Jul 07 15:50:28.858349 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:28.858357 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/3fdac274-a101-4e09-9e15-83842c752d99-data\") pod \"postgres-69fb4594cf-wrv28\" (UID: \"3fdac274-a101-4e09-9e15-83842c752d99\") " pod="opendatahub/postgres-69fb4594cf-wrv28" Jul 07 15:50:28.858749 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:28.858732 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/3fdac274-a101-4e09-9e15-83842c752d99-data\") pod \"postgres-69fb4594cf-wrv28\" (UID: \"3fdac274-a101-4e09-9e15-83842c752d99\") " pod="opendatahub/postgres-69fb4594cf-wrv28" Jul 07 15:50:28.868600 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:28.868574 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9j6gd\" (UniqueName: \"kubernetes.io/projected/3fdac274-a101-4e09-9e15-83842c752d99-kube-api-access-9j6gd\") pod \"postgres-69fb4594cf-wrv28\" (UID: \"3fdac274-a101-4e09-9e15-83842c752d99\") " pod="opendatahub/postgres-69fb4594cf-wrv28" Jul 07 15:50:29.026257 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:29.026153 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/postgres-69fb4594cf-wrv28" Jul 07 15:50:29.163110 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:29.163087 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/postgres-69fb4594cf-wrv28"] Jul 07 15:50:29.165265 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:50:29.165237 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fdac274_a101_4e09_9e15_83842c752d99.slice/crio-d65075771aab425e8462751dad97beb589efeca3731ef5036698c29cdebc99cd WatchSource:0}: Error finding container d65075771aab425e8462751dad97beb589efeca3731ef5036698c29cdebc99cd: Status 404 returned error can't find the container with id d65075771aab425e8462751dad97beb589efeca3731ef5036698c29cdebc99cd Jul 07 15:50:30.104486 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:30.104451 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/postgres-69fb4594cf-wrv28" event={"ID":"3fdac274-a101-4e09-9e15-83842c752d99","Type":"ContainerStarted","Data":"d65075771aab425e8462751dad97beb589efeca3731ef5036698c29cdebc99cd"} Jul 07 15:50:34.127183 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:34.127084 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/postgres-69fb4594cf-wrv28" event={"ID":"3fdac274-a101-4e09-9e15-83842c752d99","Type":"ContainerStarted","Data":"4ee3f382a840eceda37a34f0ac4cdb4409ab0417006c9558b6a0dbe5ca3a980e"} Jul 07 15:50:34.127544 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:34.127296 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/postgres-69fb4594cf-wrv28" Jul 07 15:50:40.162091 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:40.162063 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/postgres-69fb4594cf-wrv28" Jul 07 15:50:40.185860 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:40.185809 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/postgres-69fb4594cf-wrv28" podStartSLOduration=7.489438242 podStartE2EDuration="12.185793759s" totalImagesPullingTime="4.696355517s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:50:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:50:34.164590576 +0000 UTC m=+573.121168042" watchObservedRunningTime="2026-07-07 15:50:40.185793759 +0000 UTC m=+579.142371226" Jul 07 15:50:40.572687 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:40.572657 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:50:40.670280 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:40.670243 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:50:40.768688 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:40.768662 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5d4b67c446-7c8t8"] Jul 07 15:50:40.774768 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:40.774717 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-7c8t8" Jul 07 15:50:40.781266 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:40.781245 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-ctdxm\"" Jul 07 15:50:40.786639 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:40.786620 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-7c8t8"] Jul 07 15:50:40.865367 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:40.865335 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xt4vx\" (UniqueName: \"kubernetes.io/projected/6db5132a-0aac-4c5a-9395-4502115edd11-kube-api-access-xt4vx\") pod \"authorino-5d4b67c446-7c8t8\" (UID: \"6db5132a-0aac-4c5a-9395-4502115edd11\") " pod="kuadrant-system/authorino-5d4b67c446-7c8t8" Jul 07 15:50:40.965818 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:40.965779 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xt4vx\" (UniqueName: \"kubernetes.io/projected/6db5132a-0aac-4c5a-9395-4502115edd11-kube-api-access-xt4vx\") pod \"authorino-5d4b67c446-7c8t8\" (UID: \"6db5132a-0aac-4c5a-9395-4502115edd11\") " pod="kuadrant-system/authorino-5d4b67c446-7c8t8" Jul 07 15:50:40.978348 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:40.978321 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xt4vx\" (UniqueName: \"kubernetes.io/projected/6db5132a-0aac-4c5a-9395-4502115edd11-kube-api-access-xt4vx\") pod \"authorino-5d4b67c446-7c8t8\" (UID: \"6db5132a-0aac-4c5a-9395-4502115edd11\") " pod="kuadrant-system/authorino-5d4b67c446-7c8t8" Jul 07 15:50:41.055846 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:41.055762 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-7c8t8"] Jul 07 15:50:41.056020 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:41.056009 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-7c8t8" Jul 07 15:50:41.114694 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:41.114665 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5d78dd898-ghjvl"] Jul 07 15:50:41.124353 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:41.124320 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d78dd898-ghjvl" Jul 07 15:50:41.130238 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:41.130210 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d78dd898-ghjvl"] Jul 07 15:50:41.167773 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:41.167737 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bjmz\" (UniqueName: \"kubernetes.io/projected/50945f31-f563-44d8-8783-688e60ee7044-kube-api-access-6bjmz\") pod \"authorino-5d78dd898-ghjvl\" (UID: \"50945f31-f563-44d8-8783-688e60ee7044\") " pod="kuadrant-system/authorino-5d78dd898-ghjvl" Jul 07 15:50:41.204767 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:41.204743 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-7c8t8"] Jul 07 15:50:41.206524 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:50:41.206463 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6db5132a_0aac_4c5a_9395_4502115edd11.slice/crio-85c9db544a99b419f6d2f76c9dc6ea9131a462302dfc6dff028871b3973b52d5 WatchSource:0}: Error finding container 85c9db544a99b419f6d2f76c9dc6ea9131a462302dfc6dff028871b3973b52d5: Status 404 returned error can't find the container with id 85c9db544a99b419f6d2f76c9dc6ea9131a462302dfc6dff028871b3973b52d5 Jul 07 15:50:41.236944 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:41.236910 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d78dd898-ghjvl"] Jul 07 15:50:41.237231 ip-10-0-132-65 kubenswrapper[2565]: E0707 15:50:41.237207 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-6bjmz], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-5d78dd898-ghjvl" podUID="50945f31-f563-44d8-8783-688e60ee7044" Jul 07 15:50:41.268722 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:41.268685 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bjmz\" (UniqueName: \"kubernetes.io/projected/50945f31-f563-44d8-8783-688e60ee7044-kube-api-access-6bjmz\") pod \"authorino-5d78dd898-ghjvl\" (UID: \"50945f31-f563-44d8-8783-688e60ee7044\") " pod="kuadrant-system/authorino-5d78dd898-ghjvl" Jul 07 15:50:41.285526 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:41.285474 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-84bb5794f4-bxncs"] Jul 07 15:50:41.289219 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:41.289195 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-84bb5794f4-bxncs" Jul 07 15:50:41.290038 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:41.290018 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bjmz\" (UniqueName: \"kubernetes.io/projected/50945f31-f563-44d8-8783-688e60ee7044-kube-api-access-6bjmz\") pod \"authorino-5d78dd898-ghjvl\" (UID: \"50945f31-f563-44d8-8783-688e60ee7044\") " pod="kuadrant-system/authorino-5d78dd898-ghjvl" Jul 07 15:50:41.297810 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:41.297789 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Jul 07 15:50:41.309012 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:41.308959 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-84bb5794f4-bxncs"] Jul 07 15:50:41.369548 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:41.369504 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5f7mp\" (UniqueName: \"kubernetes.io/projected/97baad96-0f60-4c9f-84b5-7bd018f22ac9-kube-api-access-5f7mp\") pod \"authorino-84bb5794f4-bxncs\" (UID: \"97baad96-0f60-4c9f-84b5-7bd018f22ac9\") " pod="kuadrant-system/authorino-84bb5794f4-bxncs" Jul 07 15:50:41.369708 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:41.369574 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/97baad96-0f60-4c9f-84b5-7bd018f22ac9-tls-cert\") pod \"authorino-84bb5794f4-bxncs\" (UID: \"97baad96-0f60-4c9f-84b5-7bd018f22ac9\") " pod="kuadrant-system/authorino-84bb5794f4-bxncs" Jul 07 15:50:41.470398 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:41.470361 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5f7mp\" (UniqueName: \"kubernetes.io/projected/97baad96-0f60-4c9f-84b5-7bd018f22ac9-kube-api-access-5f7mp\") pod \"authorino-84bb5794f4-bxncs\" (UID: \"97baad96-0f60-4c9f-84b5-7bd018f22ac9\") " pod="kuadrant-system/authorino-84bb5794f4-bxncs" Jul 07 15:50:41.470570 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:41.470425 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/97baad96-0f60-4c9f-84b5-7bd018f22ac9-tls-cert\") pod \"authorino-84bb5794f4-bxncs\" (UID: \"97baad96-0f60-4c9f-84b5-7bd018f22ac9\") " pod="kuadrant-system/authorino-84bb5794f4-bxncs" Jul 07 15:50:41.472721 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:41.472704 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/97baad96-0f60-4c9f-84b5-7bd018f22ac9-tls-cert\") pod \"authorino-84bb5794f4-bxncs\" (UID: \"97baad96-0f60-4c9f-84b5-7bd018f22ac9\") " pod="kuadrant-system/authorino-84bb5794f4-bxncs" Jul 07 15:50:41.489221 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:41.489196 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5f7mp\" (UniqueName: \"kubernetes.io/projected/97baad96-0f60-4c9f-84b5-7bd018f22ac9-kube-api-access-5f7mp\") pod \"authorino-84bb5794f4-bxncs\" (UID: \"97baad96-0f60-4c9f-84b5-7bd018f22ac9\") " pod="kuadrant-system/authorino-84bb5794f4-bxncs" Jul 07 15:50:41.605598 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:41.605562 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-84bb5794f4-bxncs" Jul 07 15:50:41.759777 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:41.759752 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-84bb5794f4-bxncs"] Jul 07 15:50:42.165561 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:42.165524 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-7c8t8" event={"ID":"6db5132a-0aac-4c5a-9395-4502115edd11","Type":"ContainerStarted","Data":"1a17b02dbef49d7466deb737934af174d68efe3596ba13d29e874fc9d077f3ce"} Jul 07 15:50:42.165686 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:42.165577 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-7c8t8" event={"ID":"6db5132a-0aac-4c5a-9395-4502115edd11","Type":"ContainerStarted","Data":"85c9db544a99b419f6d2f76c9dc6ea9131a462302dfc6dff028871b3973b52d5"} Jul 07 15:50:42.165686 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:42.165591 2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/authorino-5d4b67c446-7c8t8" podUID="6db5132a-0aac-4c5a-9395-4502115edd11" containerName="authorino" containerID="cri-o://1a17b02dbef49d7466deb737934af174d68efe3596ba13d29e874fc9d077f3ce" gracePeriod=30 Jul 07 15:50:42.166839 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:42.166808 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-84bb5794f4-bxncs" event={"ID":"97baad96-0f60-4c9f-84b5-7bd018f22ac9","Type":"ContainerStarted","Data":"2629dfe6e5879696ec4f0c815b6ff32fe559a2b637c8c108b89d1ea6508c8042"} Jul 07 15:50:42.166943 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:42.166818 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d78dd898-ghjvl" Jul 07 15:50:42.178558 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:42.178535 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d78dd898-ghjvl" Jul 07 15:50:42.196051 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:42.196007 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5d4b67c446-7c8t8" podStartSLOduration=1.6492374939999999 podStartE2EDuration="2.19599227s" totalImagesPullingTime="546.754776ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:50:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:50:42.195103816 +0000 UTC m=+581.151681295" watchObservedRunningTime="2026-07-07 15:50:42.19599227 +0000 UTC m=+581.152569736" Jul 07 15:50:42.277611 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:42.277582 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6bjmz\" (UniqueName: \"kubernetes.io/projected/50945f31-f563-44d8-8783-688e60ee7044-kube-api-access-6bjmz\") pod \"50945f31-f563-44d8-8783-688e60ee7044\" (UID: \"50945f31-f563-44d8-8783-688e60ee7044\") " Jul 07 15:50:42.279786 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:42.279759 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50945f31-f563-44d8-8783-688e60ee7044-kube-api-access-6bjmz" (OuterVolumeSpecName: "kube-api-access-6bjmz") pod "50945f31-f563-44d8-8783-688e60ee7044" (UID: "50945f31-f563-44d8-8783-688e60ee7044"). InnerVolumeSpecName "kube-api-access-6bjmz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 15:50:42.379238 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:42.379161 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6bjmz\" (UniqueName: \"kubernetes.io/projected/50945f31-f563-44d8-8783-688e60ee7044-kube-api-access-6bjmz\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 07 15:50:42.454003 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:42.453968 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-7c8t8" Jul 07 15:50:42.581096 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:42.581059 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xt4vx\" (UniqueName: \"kubernetes.io/projected/6db5132a-0aac-4c5a-9395-4502115edd11-kube-api-access-xt4vx\") pod \"6db5132a-0aac-4c5a-9395-4502115edd11\" (UID: \"6db5132a-0aac-4c5a-9395-4502115edd11\") " Jul 07 15:50:42.583120 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:42.583097 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6db5132a-0aac-4c5a-9395-4502115edd11-kube-api-access-xt4vx" (OuterVolumeSpecName: "kube-api-access-xt4vx") pod "6db5132a-0aac-4c5a-9395-4502115edd11" (UID: "6db5132a-0aac-4c5a-9395-4502115edd11"). InnerVolumeSpecName "kube-api-access-xt4vx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 15:50:42.682336 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:42.682255 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xt4vx\" (UniqueName: \"kubernetes.io/projected/6db5132a-0aac-4c5a-9395-4502115edd11-kube-api-access-xt4vx\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 07 15:50:43.172999 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:43.172962 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-84bb5794f4-bxncs" event={"ID":"97baad96-0f60-4c9f-84b5-7bd018f22ac9","Type":"ContainerStarted","Data":"bcfbbdf83b1e3105820e5aa7dbd623d3bc1911baeca6e02511112b97a4852d10"} Jul 07 15:50:43.174258 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:43.174226 2565 generic.go:356] "Generic (PLEG): container finished" podID="6db5132a-0aac-4c5a-9395-4502115edd11" containerID="1a17b02dbef49d7466deb737934af174d68efe3596ba13d29e874fc9d077f3ce" exitCode=0 Jul 07 15:50:43.174369 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:43.174267 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-7c8t8" Jul 07 15:50:43.174369 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:43.174301 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-7c8t8" event={"ID":"6db5132a-0aac-4c5a-9395-4502115edd11","Type":"ContainerDied","Data":"1a17b02dbef49d7466deb737934af174d68efe3596ba13d29e874fc9d077f3ce"} Jul 07 15:50:43.174369 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:43.174332 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-7c8t8" event={"ID":"6db5132a-0aac-4c5a-9395-4502115edd11","Type":"ContainerDied","Data":"85c9db544a99b419f6d2f76c9dc6ea9131a462302dfc6dff028871b3973b52d5"} Jul 07 15:50:43.174369 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:43.174354 2565 scope.go:117] "RemoveContainer" containerID="1a17b02dbef49d7466deb737934af174d68efe3596ba13d29e874fc9d077f3ce" Jul 07 15:50:43.174537 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:43.174463 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d78dd898-ghjvl" Jul 07 15:50:43.183463 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:43.183447 2565 scope.go:117] "RemoveContainer" containerID="1a17b02dbef49d7466deb737934af174d68efe3596ba13d29e874fc9d077f3ce" Jul 07 15:50:43.183808 ip-10-0-132-65 kubenswrapper[2565]: E0707 15:50:43.183777 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a17b02dbef49d7466deb737934af174d68efe3596ba13d29e874fc9d077f3ce\": container with ID starting with 1a17b02dbef49d7466deb737934af174d68efe3596ba13d29e874fc9d077f3ce not found: ID does not exist" containerID="1a17b02dbef49d7466deb737934af174d68efe3596ba13d29e874fc9d077f3ce" Jul 07 15:50:43.183971 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:43.183809 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a17b02dbef49d7466deb737934af174d68efe3596ba13d29e874fc9d077f3ce"} err="failed to get container status \"1a17b02dbef49d7466deb737934af174d68efe3596ba13d29e874fc9d077f3ce\": rpc error: code = NotFound desc = could not find container \"1a17b02dbef49d7466deb737934af174d68efe3596ba13d29e874fc9d077f3ce\": container with ID starting with 1a17b02dbef49d7466deb737934af174d68efe3596ba13d29e874fc9d077f3ce not found: ID does not exist" Jul 07 15:50:43.196651 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:43.196609 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-84bb5794f4-bxncs" podStartSLOduration=1.8365004919999999 podStartE2EDuration="2.196595469s" totalImagesPullingTime="360.094977ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:50:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:50:43.196091979 +0000 UTC m=+582.152669458" watchObservedRunningTime="2026-07-07 15:50:43.196595469 +0000 UTC m=+582.153172936" Jul 07 15:50:43.217375 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:43.217344 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-7c8t8"] Jul 07 15:50:43.224951 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:43.224919 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-7c8t8"] Jul 07 15:50:43.277065 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:43.276981 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d78dd898-ghjvl"] Jul 07 15:50:43.278487 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:43.278465 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5d78dd898-ghjvl"] Jul 07 15:50:43.660694 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:43.660661 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="50945f31-f563-44d8-8783-688e60ee7044" path="/var/lib/kubelet/pods/50945f31-f563-44d8-8783-688e60ee7044/volumes" Jul 07 15:50:43.660899 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:43.660887 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6db5132a-0aac-4c5a-9395-4502115edd11" path="/var/lib/kubelet/pods/6db5132a-0aac-4c5a-9395-4502115edd11/volumes" Jul 07 15:50:48.013360 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:48.013323 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:50:48.029951 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:48.029922 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-66c5987586-9sx22"] Jul 07 15:50:48.030274 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:48.030262 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6db5132a-0aac-4c5a-9395-4502115edd11" containerName="authorino" Jul 07 15:50:48.030316 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:48.030275 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="6db5132a-0aac-4c5a-9395-4502115edd11" containerName="authorino" Jul 07 15:50:48.030349 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:48.030334 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="6db5132a-0aac-4c5a-9395-4502115edd11" containerName="authorino" Jul 07 15:50:48.046408 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:48.046381 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-66c5987586-9sx22" Jul 07 15:50:48.051062 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:48.051024 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 07 15:50:48.051195 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:48.051062 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-rdqhj\"" Jul 07 15:50:48.052178 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:48.052158 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 07 15:50:48.056870 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:48.056840 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-66c5987586-9sx22"] Jul 07 15:50:48.131774 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:48.131728 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ch6x8\" (UniqueName: \"kubernetes.io/projected/644bae7a-baea-4372-8104-42f3705e6d37-kube-api-access-ch6x8\") pod \"maas-controller-66c5987586-9sx22\" (UID: \"644bae7a-baea-4372-8104-42f3705e6d37\") " pod="opendatahub/maas-controller-66c5987586-9sx22" Jul 07 15:50:48.131956 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:48.131792 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/644bae7a-baea-4372-8104-42f3705e6d37-webhook-cert\") pod \"maas-controller-66c5987586-9sx22\" (UID: \"644bae7a-baea-4372-8104-42f3705e6d37\") " pod="opendatahub/maas-controller-66c5987586-9sx22" Jul 07 15:50:48.146113 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:48.146080 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:50:48.232462 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:48.232429 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ch6x8\" (UniqueName: \"kubernetes.io/projected/644bae7a-baea-4372-8104-42f3705e6d37-kube-api-access-ch6x8\") pod \"maas-controller-66c5987586-9sx22\" (UID: \"644bae7a-baea-4372-8104-42f3705e6d37\") " pod="opendatahub/maas-controller-66c5987586-9sx22" Jul 07 15:50:48.232683 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:48.232476 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/644bae7a-baea-4372-8104-42f3705e6d37-webhook-cert\") pod \"maas-controller-66c5987586-9sx22\" (UID: \"644bae7a-baea-4372-8104-42f3705e6d37\") " pod="opendatahub/maas-controller-66c5987586-9sx22" Jul 07 15:50:48.234931 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:48.234911 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/644bae7a-baea-4372-8104-42f3705e6d37-webhook-cert\") pod \"maas-controller-66c5987586-9sx22\" (UID: \"644bae7a-baea-4372-8104-42f3705e6d37\") " pod="opendatahub/maas-controller-66c5987586-9sx22" Jul 07 15:50:48.242855 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:48.242832 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:50:48.244735 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:48.244714 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ch6x8\" (UniqueName: \"kubernetes.io/projected/644bae7a-baea-4372-8104-42f3705e6d37-kube-api-access-ch6x8\") pod \"maas-controller-66c5987586-9sx22\" (UID: \"644bae7a-baea-4372-8104-42f3705e6d37\") " pod="opendatahub/maas-controller-66c5987586-9sx22" Jul 07 15:50:48.272248 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:48.272184 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-586d4c4bbb-fpxn2"] Jul 07 15:50:48.274587 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:48.274564 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-586d4c4bbb-fpxn2" Jul 07 15:50:48.294338 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:48.294310 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-586d4c4bbb-fpxn2"] Jul 07 15:50:48.333737 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:48.333706 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/bb501c65-31db-438e-b944-3f553af1b836-webhook-cert\") pod \"maas-controller-586d4c4bbb-fpxn2\" (UID: \"bb501c65-31db-438e-b944-3f553af1b836\") " pod="opendatahub/maas-controller-586d4c4bbb-fpxn2" Jul 07 15:50:48.333924 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:48.333755 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f6hm4\" (UniqueName: \"kubernetes.io/projected/bb501c65-31db-438e-b944-3f553af1b836-kube-api-access-f6hm4\") pod \"maas-controller-586d4c4bbb-fpxn2\" (UID: \"bb501c65-31db-438e-b944-3f553af1b836\") " pod="opendatahub/maas-controller-586d4c4bbb-fpxn2" Jul 07 15:50:48.358361 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:48.358336 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-66c5987586-9sx22" Jul 07 15:50:48.434537 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:48.434484 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/bb501c65-31db-438e-b944-3f553af1b836-webhook-cert\") pod \"maas-controller-586d4c4bbb-fpxn2\" (UID: \"bb501c65-31db-438e-b944-3f553af1b836\") " pod="opendatahub/maas-controller-586d4c4bbb-fpxn2" Jul 07 15:50:48.434687 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:48.434580 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6hm4\" (UniqueName: \"kubernetes.io/projected/bb501c65-31db-438e-b944-3f553af1b836-kube-api-access-f6hm4\") pod \"maas-controller-586d4c4bbb-fpxn2\" (UID: \"bb501c65-31db-438e-b944-3f553af1b836\") " pod="opendatahub/maas-controller-586d4c4bbb-fpxn2" Jul 07 15:50:48.436796 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:48.436775 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/bb501c65-31db-438e-b944-3f553af1b836-webhook-cert\") pod \"maas-controller-586d4c4bbb-fpxn2\" (UID: \"bb501c65-31db-438e-b944-3f553af1b836\") " pod="opendatahub/maas-controller-586d4c4bbb-fpxn2" Jul 07 15:50:48.451850 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:48.451815 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6hm4\" (UniqueName: \"kubernetes.io/projected/bb501c65-31db-438e-b944-3f553af1b836-kube-api-access-f6hm4\") pod \"maas-controller-586d4c4bbb-fpxn2\" (UID: \"bb501c65-31db-438e-b944-3f553af1b836\") " pod="opendatahub/maas-controller-586d4c4bbb-fpxn2" Jul 07 15:50:48.500189 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:48.500085 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-66c5987586-9sx22"] Jul 07 15:50:48.502636 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:50:48.502604 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod644bae7a_baea_4372_8104_42f3705e6d37.slice/crio-de3a697109a740004b14afadd8520a64427c1ea64f21dcecee0052d53e01c313 WatchSource:0}: Error finding container de3a697109a740004b14afadd8520a64427c1ea64f21dcecee0052d53e01c313: Status 404 returned error can't find the container with id de3a697109a740004b14afadd8520a64427c1ea64f21dcecee0052d53e01c313 Jul 07 15:50:48.585656 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:48.585618 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-586d4c4bbb-fpxn2" Jul 07 15:50:48.718228 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:48.718203 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-586d4c4bbb-fpxn2"] Jul 07 15:50:48.720589 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:50:48.720560 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb501c65_31db_438e_b944_3f553af1b836.slice/crio-80b7091873a887498f02c25a427f21787edeab9d0c764cf918f5be897522a604 WatchSource:0}: Error finding container 80b7091873a887498f02c25a427f21787edeab9d0c764cf918f5be897522a604: Status 404 returned error can't find the container with id 80b7091873a887498f02c25a427f21787edeab9d0c764cf918f5be897522a604 Jul 07 15:50:49.204118 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:49.204079 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-586d4c4bbb-fpxn2" event={"ID":"bb501c65-31db-438e-b944-3f553af1b836","Type":"ContainerStarted","Data":"80b7091873a887498f02c25a427f21787edeab9d0c764cf918f5be897522a604"} Jul 07 15:50:49.205403 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:49.205375 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-66c5987586-9sx22" event={"ID":"644bae7a-baea-4372-8104-42f3705e6d37","Type":"ContainerStarted","Data":"de3a697109a740004b14afadd8520a64427c1ea64f21dcecee0052d53e01c313"} Jul 07 15:50:51.221553 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:51.221515 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-586d4c4bbb-fpxn2" event={"ID":"bb501c65-31db-438e-b944-3f553af1b836","Type":"ContainerStarted","Data":"d928d64ce7bdb24de776344f33a710b4ba17f252ae7545855c7dd03092f31102"} Jul 07 15:50:51.221994 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:51.221615 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-586d4c4bbb-fpxn2" Jul 07 15:50:51.223370 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:51.223341 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-66c5987586-9sx22" event={"ID":"644bae7a-baea-4372-8104-42f3705e6d37","Type":"ContainerStarted","Data":"3b17c64d453336b1a60a4cda16b0096a4cc0dc215ad4fffd300891b2eb6670cb"} Jul 07 15:50:51.223529 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:51.223511 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-66c5987586-9sx22" Jul 07 15:50:51.251280 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:51.251226 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-586d4c4bbb-fpxn2" podStartSLOduration=0.848165016 podStartE2EDuration="3.251208877s" totalImagesPullingTime="2.403043861s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:50:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:50:51.249810023 +0000 UTC m=+590.206387492" watchObservedRunningTime="2026-07-07 15:50:51.251208877 +0000 UTC m=+590.207786343" Jul 07 15:50:51.274578 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:51.274525 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-66c5987586-9sx22" podStartSLOduration=0.660098988 podStartE2EDuration="3.274506137s" totalImagesPullingTime="2.614407149s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:50:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:50:51.271007595 +0000 UTC m=+590.227585123" watchObservedRunningTime="2026-07-07 15:50:51.274506137 +0000 UTC m=+590.231083602" Jul 07 15:50:51.384394 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:51.384303 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:50:51.501206 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:50:51.501171 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:51:01.500170 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:51:01.500142 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-s9kl8_62da0165-d32c-4bba-9844-a49154d765e5/console-operator/1.log" Jul 07 15:51:01.504787 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:51:01.504765 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-942xh_31ff16f0-6054-4eba-942a-b07a5a7afac7/ovn-acl-logging/0.log" Jul 07 15:51:01.506945 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:51:01.506918 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-s9kl8_62da0165-d32c-4bba-9844-a49154d765e5/console-operator/1.log" Jul 07 15:51:01.517229 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:51:01.517208 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-942xh_31ff16f0-6054-4eba-942a-b07a5a7afac7/ovn-acl-logging/0.log" Jul 07 15:51:02.236476 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:51:02.236448 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-586d4c4bbb-fpxn2" Jul 07 15:51:02.236667 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:51:02.236539 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-66c5987586-9sx22" Jul 07 15:51:02.342747 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:51:02.342718 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-66c5987586-9sx22"] Jul 07 15:51:02.342905 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:51:02.342885 2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-controller-66c5987586-9sx22" podUID="644bae7a-baea-4372-8104-42f3705e6d37" containerName="manager" containerID="cri-o://3b17c64d453336b1a60a4cda16b0096a4cc0dc215ad4fffd300891b2eb6670cb" gracePeriod=30 Jul 07 15:51:02.587951 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:51:02.587926 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-66c5987586-9sx22" Jul 07 15:51:02.764966 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:51:02.764879 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/644bae7a-baea-4372-8104-42f3705e6d37-webhook-cert\") pod \"644bae7a-baea-4372-8104-42f3705e6d37\" (UID: \"644bae7a-baea-4372-8104-42f3705e6d37\") " Jul 07 15:51:02.765114 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:51:02.764964 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ch6x8\" (UniqueName: \"kubernetes.io/projected/644bae7a-baea-4372-8104-42f3705e6d37-kube-api-access-ch6x8\") pod \"644bae7a-baea-4372-8104-42f3705e6d37\" (UID: \"644bae7a-baea-4372-8104-42f3705e6d37\") " Jul 07 15:51:02.767088 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:51:02.767058 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/644bae7a-baea-4372-8104-42f3705e6d37-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "644bae7a-baea-4372-8104-42f3705e6d37" (UID: "644bae7a-baea-4372-8104-42f3705e6d37"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 15:51:02.767196 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:51:02.767112 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/644bae7a-baea-4372-8104-42f3705e6d37-kube-api-access-ch6x8" (OuterVolumeSpecName: "kube-api-access-ch6x8") pod "644bae7a-baea-4372-8104-42f3705e6d37" (UID: "644bae7a-baea-4372-8104-42f3705e6d37"). InnerVolumeSpecName "kube-api-access-ch6x8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 15:51:02.865644 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:51:02.865608 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ch6x8\" (UniqueName: \"kubernetes.io/projected/644bae7a-baea-4372-8104-42f3705e6d37-kube-api-access-ch6x8\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 07 15:51:02.865644 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:51:02.865639 2565 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/644bae7a-baea-4372-8104-42f3705e6d37-webhook-cert\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 07 15:51:03.278683 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:51:03.278646 2565 generic.go:356] "Generic (PLEG): container finished" podID="644bae7a-baea-4372-8104-42f3705e6d37" containerID="3b17c64d453336b1a60a4cda16b0096a4cc0dc215ad4fffd300891b2eb6670cb" exitCode=0 Jul 07 15:51:03.278830 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:51:03.278699 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-66c5987586-9sx22" Jul 07 15:51:03.278830 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:51:03.278696 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-66c5987586-9sx22" event={"ID":"644bae7a-baea-4372-8104-42f3705e6d37","Type":"ContainerDied","Data":"3b17c64d453336b1a60a4cda16b0096a4cc0dc215ad4fffd300891b2eb6670cb"} Jul 07 15:51:03.278931 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:51:03.278828 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-66c5987586-9sx22" event={"ID":"644bae7a-baea-4372-8104-42f3705e6d37","Type":"ContainerDied","Data":"de3a697109a740004b14afadd8520a64427c1ea64f21dcecee0052d53e01c313"} Jul 07 15:51:03.278931 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:51:03.278857 2565 scope.go:117] "RemoveContainer" containerID="3b17c64d453336b1a60a4cda16b0096a4cc0dc215ad4fffd300891b2eb6670cb" Jul 07 15:51:03.287761 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:51:03.287744 2565 scope.go:117] "RemoveContainer" containerID="3b17c64d453336b1a60a4cda16b0096a4cc0dc215ad4fffd300891b2eb6670cb" Jul 07 15:51:03.288008 ip-10-0-132-65 kubenswrapper[2565]: E0707 15:51:03.287990 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b17c64d453336b1a60a4cda16b0096a4cc0dc215ad4fffd300891b2eb6670cb\": container with ID starting with 3b17c64d453336b1a60a4cda16b0096a4cc0dc215ad4fffd300891b2eb6670cb not found: ID does not exist" containerID="3b17c64d453336b1a60a4cda16b0096a4cc0dc215ad4fffd300891b2eb6670cb" Jul 07 15:51:03.288076 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:51:03.288017 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b17c64d453336b1a60a4cda16b0096a4cc0dc215ad4fffd300891b2eb6670cb"} err="failed to get container status \"3b17c64d453336b1a60a4cda16b0096a4cc0dc215ad4fffd300891b2eb6670cb\": rpc error: code = NotFound desc = could not find container \"3b17c64d453336b1a60a4cda16b0096a4cc0dc215ad4fffd300891b2eb6670cb\": container with ID starting with 3b17c64d453336b1a60a4cda16b0096a4cc0dc215ad4fffd300891b2eb6670cb not found: ID does not exist" Jul 07 15:51:03.307279 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:51:03.307257 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-66c5987586-9sx22"] Jul 07 15:51:03.314620 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:51:03.314601 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-66c5987586-9sx22"] Jul 07 15:51:03.660859 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:51:03.660826 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="644bae7a-baea-4372-8104-42f3705e6d37" path="/var/lib/kubelet/pods/644bae7a-baea-4372-8104-42f3705e6d37/volumes" Jul 07 15:51:23.475618 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:51:23.475576 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:51:23.970479 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:51:23.970448 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:51:24.006586 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:51:24.006550 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-5f874bfc85-qc79p"] Jul 07 15:51:24.006922 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:51:24.006910 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="644bae7a-baea-4372-8104-42f3705e6d37" containerName="manager" Jul 07 15:51:24.006966 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:51:24.006925 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="644bae7a-baea-4372-8104-42f3705e6d37" containerName="manager" Jul 07 15:51:24.006999 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:51:24.006987 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="644bae7a-baea-4372-8104-42f3705e6d37" containerName="manager" Jul 07 15:51:24.011332 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:51:24.011312 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-5f874bfc85-qc79p" Jul 07 15:51:24.018189 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:51:24.017947 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-serving-cert\"" Jul 07 15:51:24.026997 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:51:24.026973 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-5f874bfc85-qc79p"] Jul 07 15:51:24.043747 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:51:24.043717 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5hgx\" (UniqueName: \"kubernetes.io/projected/5ae76816-ef6b-4546-9b65-571e4a89d035-kube-api-access-h5hgx\") pod \"maas-api-5f874bfc85-qc79p\" (UID: \"5ae76816-ef6b-4546-9b65-571e4a89d035\") " pod="opendatahub/maas-api-5f874bfc85-qc79p" Jul 07 15:51:24.043887 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:51:24.043763 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ae76816-ef6b-4546-9b65-571e4a89d035-openshift-service-ca\") pod \"maas-api-5f874bfc85-qc79p\" (UID: \"5ae76816-ef6b-4546-9b65-571e4a89d035\") " pod="opendatahub/maas-api-5f874bfc85-qc79p" Jul 07 15:51:24.043887 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:51:24.043854 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5ae76816-ef6b-4546-9b65-571e4a89d035-maas-api-tls\") pod \"maas-api-5f874bfc85-qc79p\" (UID: \"5ae76816-ef6b-4546-9b65-571e4a89d035\") " pod="opendatahub/maas-api-5f874bfc85-qc79p" Jul 07 15:51:24.145033 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:51:24.144997 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5ae76816-ef6b-4546-9b65-571e4a89d035-maas-api-tls\") pod \"maas-api-5f874bfc85-qc79p\" (UID: \"5ae76816-ef6b-4546-9b65-571e4a89d035\") " pod="opendatahub/maas-api-5f874bfc85-qc79p" Jul 07 15:51:24.145255 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:51:24.145072 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5hgx\" (UniqueName: \"kubernetes.io/projected/5ae76816-ef6b-4546-9b65-571e4a89d035-kube-api-access-h5hgx\") pod \"maas-api-5f874bfc85-qc79p\" (UID: \"5ae76816-ef6b-4546-9b65-571e4a89d035\") " pod="opendatahub/maas-api-5f874bfc85-qc79p" Jul 07 15:51:24.145255 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:51:24.145108 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ae76816-ef6b-4546-9b65-571e4a89d035-openshift-service-ca\") pod \"maas-api-5f874bfc85-qc79p\" (UID: \"5ae76816-ef6b-4546-9b65-571e4a89d035\") " pod="opendatahub/maas-api-5f874bfc85-qc79p" Jul 07 15:51:24.145877 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:51:24.145850 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ae76816-ef6b-4546-9b65-571e4a89d035-openshift-service-ca\") pod \"maas-api-5f874bfc85-qc79p\" (UID: \"5ae76816-ef6b-4546-9b65-571e4a89d035\") " pod="opendatahub/maas-api-5f874bfc85-qc79p" Jul 07 15:51:24.147485 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:51:24.147465 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5ae76816-ef6b-4546-9b65-571e4a89d035-maas-api-tls\") pod \"maas-api-5f874bfc85-qc79p\" (UID: \"5ae76816-ef6b-4546-9b65-571e4a89d035\") " pod="opendatahub/maas-api-5f874bfc85-qc79p" Jul 07 15:51:24.159824 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:51:24.159800 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5hgx\" (UniqueName: \"kubernetes.io/projected/5ae76816-ef6b-4546-9b65-571e4a89d035-kube-api-access-h5hgx\") pod \"maas-api-5f874bfc85-qc79p\" (UID: \"5ae76816-ef6b-4546-9b65-571e4a89d035\") " pod="opendatahub/maas-api-5f874bfc85-qc79p" Jul 07 15:51:24.194197 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:51:24.194166 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:51:24.322430 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:51:24.322402 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-5f874bfc85-qc79p" Jul 07 15:51:24.500262 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:51:24.500238 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-5f874bfc85-qc79p"] Jul 07 15:51:24.503010 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:51:24.502980 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ae76816_ef6b_4546_9b65_571e4a89d035.slice/crio-83cb30d8be030111c5b9410c5d6e280315b36b4b8a3c8c73528aa2fc25944f46 WatchSource:0}: Error finding container 83cb30d8be030111c5b9410c5d6e280315b36b4b8a3c8c73528aa2fc25944f46: Status 404 returned error can't find the container with id 83cb30d8be030111c5b9410c5d6e280315b36b4b8a3c8c73528aa2fc25944f46 Jul 07 15:51:24.769287 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:51:24.769208 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:51:24.865063 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:51:24.865019 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:51:25.390563 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:51:25.390520 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-5f874bfc85-qc79p" event={"ID":"5ae76816-ef6b-4546-9b65-571e4a89d035","Type":"ContainerStarted","Data":"83cb30d8be030111c5b9410c5d6e280315b36b4b8a3c8c73528aa2fc25944f46"} Jul 07 15:51:27.401941 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:51:27.401901 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-5f874bfc85-qc79p" event={"ID":"5ae76816-ef6b-4546-9b65-571e4a89d035","Type":"ContainerStarted","Data":"948357e90de8647786c927a6ba1fcc639f365a72b87df40ff3fb9af221c572a8"} Jul 07 15:51:27.402437 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:51:27.402028 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-5f874bfc85-qc79p" Jul 07 15:51:27.428892 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:51:27.428844 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-5f874bfc85-qc79p" podStartSLOduration=2.214970632 podStartE2EDuration="4.428827868s" totalImagesPullingTime="2.213857236s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:51:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:51:27.427693819 +0000 UTC m=+626.384271285" watchObservedRunningTime="2026-07-07 15:51:27.428827868 +0000 UTC m=+626.385405335" Jul 07 15:51:30.165751 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:51:30.165715 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:51:32.427603 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:51:32.427518 2565 generic.go:356] "Generic (PLEG): container finished" podID="ea743bfe-7a54-46a1-8f15-cb6e93ee0f98" containerID="1e42d0d45f890b584a3b5682c03d9b7310633f85492a5389f2da5c2b33334539" exitCode=0 Jul 07 15:51:32.427603 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:51:32.427587 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-tqwxj" event={"ID":"ea743bfe-7a54-46a1-8f15-cb6e93ee0f98","Type":"ContainerDied","Data":"1e42d0d45f890b584a3b5682c03d9b7310633f85492a5389f2da5c2b33334539"} Jul 07 15:51:32.428019 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:51:32.427952 2565 scope.go:117] "RemoveContainer" containerID="1e42d0d45f890b584a3b5682c03d9b7310633f85492a5389f2da5c2b33334539" Jul 07 15:51:32.429012 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:51:32.428982 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 07 15:51:32.429128 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:51:32.429065 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 07 15:51:32.429128 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:51:32.429112 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 07 15:51:32.792821 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:51:32.792740 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:51:33.412351 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:51:33.412325 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-5f874bfc85-qc79p" Jul 07 15:51:33.434268 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:51:33.434231 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-tqwxj" event={"ID":"ea743bfe-7a54-46a1-8f15-cb6e93ee0f98","Type":"ContainerStarted","Data":"823c5dabd8b525dd4406e9ff4b41c3d4b7d7915b90ad7351aec6a805d1148f5f"} Jul 07 15:51:33.883618 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:51:33.883578 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-tqwxj" Jul 07 15:51:33.883929 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:51:33.883909 2565 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5896d4dd49-tqwxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body= Jul 07 15:51:33.883983 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:51:33.883955 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-tqwxj" podUID="ea743bfe-7a54-46a1-8f15-cb6e93ee0f98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" Jul 07 15:51:34.404721 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:51:34.404687 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:51:34.505130 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:51:34.505103 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:51:34.882391 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:51:34.882345 2565 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5896d4dd49-tqwxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body= Jul 07 15:51:34.882633 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:51:34.882414 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-tqwxj" podUID="ea743bfe-7a54-46a1-8f15-cb6e93ee0f98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" Jul 07 15:51:35.883362 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:51:35.883318 2565 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5896d4dd49-tqwxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body= Jul 07 15:51:35.883992 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:51:35.883955 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-tqwxj" podUID="ea743bfe-7a54-46a1-8f15-cb6e93ee0f98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" Jul 07 15:51:36.882897 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:51:36.882859 2565 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5896d4dd49-tqwxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body= Jul 07 15:51:36.883185 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:51:36.883153 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-tqwxj" podUID="ea743bfe-7a54-46a1-8f15-cb6e93ee0f98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" Jul 07 15:51:36.976515 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:51:36.973393 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-tqwxj" Jul 07 15:51:37.887081 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:51:37.887050 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-tqwxj" Jul 07 15:51:37.888157 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:51:37.888134 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-tqwxj" Jul 07 15:51:42.668486 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:51:42.668448 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:51:51.147547 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:51:51.147509 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:51:58.715395 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:51:58.715362 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-pcsd8"] Jul 07 15:51:58.717859 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:51:58.717843 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-pcsd8" Jul 07 15:51:58.721866 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:51:58.721845 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-4zfhc\"" Jul 07 15:51:58.722002 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:51:58.721899 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 07 15:51:58.722002 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:51:58.721903 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Jul 07 15:51:58.722002 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:51:58.721993 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 07 15:51:58.734071 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:51:58.734043 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-pcsd8"] Jul 07 15:51:58.754581 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:51:58.754558 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:51:58.776795 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:51:58.776765 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ed802748-bb8a-410e-bc9b-c342536171c3-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-pcsd8\" (UID: \"ed802748-bb8a-410e-bc9b-c342536171c3\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-pcsd8" Jul 07 15:51:58.777089 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:51:58.776801 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed802748-bb8a-410e-bc9b-c342536171c3-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-pcsd8\" (UID: \"ed802748-bb8a-410e-bc9b-c342536171c3\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-pcsd8" Jul 07 15:51:58.777089 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:51:58.776836 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6d7rc\" (UniqueName: \"kubernetes.io/projected/ed802748-bb8a-410e-bc9b-c342536171c3-kube-api-access-6d7rc\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-pcsd8\" (UID: \"ed802748-bb8a-410e-bc9b-c342536171c3\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-pcsd8" Jul 07 15:51:58.777089 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:51:58.776879 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed802748-bb8a-410e-bc9b-c342536171c3-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-pcsd8\" (UID: \"ed802748-bb8a-410e-bc9b-c342536171c3\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-pcsd8" Jul 07 15:51:58.777089 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:51:58.776903 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed802748-bb8a-410e-bc9b-c342536171c3-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-pcsd8\" (UID: \"ed802748-bb8a-410e-bc9b-c342536171c3\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-pcsd8" Jul 07 15:51:58.777089 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:51:58.776970 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed802748-bb8a-410e-bc9b-c342536171c3-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-pcsd8\" (UID: \"ed802748-bb8a-410e-bc9b-c342536171c3\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-pcsd8" Jul 07 15:51:58.878419 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:51:58.878384 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ed802748-bb8a-410e-bc9b-c342536171c3-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-pcsd8\" (UID: \"ed802748-bb8a-410e-bc9b-c342536171c3\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-pcsd8" Jul 07 15:51:58.878419 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:51:58.878420 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed802748-bb8a-410e-bc9b-c342536171c3-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-pcsd8\" (UID: \"ed802748-bb8a-410e-bc9b-c342536171c3\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-pcsd8" Jul 07 15:51:58.878704 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:51:58.878448 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6d7rc\" (UniqueName: \"kubernetes.io/projected/ed802748-bb8a-410e-bc9b-c342536171c3-kube-api-access-6d7rc\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-pcsd8\" (UID: \"ed802748-bb8a-410e-bc9b-c342536171c3\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-pcsd8" Jul 07 15:51:58.878704 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:51:58.878549 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed802748-bb8a-410e-bc9b-c342536171c3-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-pcsd8\" (UID: \"ed802748-bb8a-410e-bc9b-c342536171c3\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-pcsd8" Jul 07 15:51:58.878704 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:51:58.878574 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed802748-bb8a-410e-bc9b-c342536171c3-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-pcsd8\" (UID: \"ed802748-bb8a-410e-bc9b-c342536171c3\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-pcsd8" Jul 07 15:51:58.878704 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:51:58.878625 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed802748-bb8a-410e-bc9b-c342536171c3-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-pcsd8\" (UID: \"ed802748-bb8a-410e-bc9b-c342536171c3\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-pcsd8" Jul 07 15:51:58.878938 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:51:58.878924 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed802748-bb8a-410e-bc9b-c342536171c3-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-pcsd8\" (UID: \"ed802748-bb8a-410e-bc9b-c342536171c3\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-pcsd8" Jul 07 15:51:58.878983 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:51:58.878919 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ed802748-bb8a-410e-bc9b-c342536171c3-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-pcsd8\" (UID: \"ed802748-bb8a-410e-bc9b-c342536171c3\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-pcsd8" Jul 07 15:51:58.879033 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:51:58.878979 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed802748-bb8a-410e-bc9b-c342536171c3-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-pcsd8\" (UID: \"ed802748-bb8a-410e-bc9b-c342536171c3\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-pcsd8" Jul 07 15:51:58.880947 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:51:58.880920 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed802748-bb8a-410e-bc9b-c342536171c3-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-pcsd8\" (UID: \"ed802748-bb8a-410e-bc9b-c342536171c3\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-pcsd8" Jul 07 15:51:58.881160 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:51:58.881142 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed802748-bb8a-410e-bc9b-c342536171c3-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-pcsd8\" (UID: \"ed802748-bb8a-410e-bc9b-c342536171c3\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-pcsd8" Jul 07 15:51:58.887769 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:51:58.887749 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6d7rc\" (UniqueName: \"kubernetes.io/projected/ed802748-bb8a-410e-bc9b-c342536171c3-kube-api-access-6d7rc\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-pcsd8\" (UID: \"ed802748-bb8a-410e-bc9b-c342536171c3\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-pcsd8" Jul 07 15:51:59.028866 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:51:59.028782 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-pcsd8" Jul 07 15:51:59.164791 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:51:59.164764 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-pcsd8"] Jul 07 15:51:59.166538 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:51:59.166508 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded802748_bb8a_410e_bc9b_c342536171c3.slice/crio-29b670ca52e3a6622e45bdbdb35563f9d746be9cbd65b6f5d96612473a22697f WatchSource:0}: Error finding container 29b670ca52e3a6622e45bdbdb35563f9d746be9cbd65b6f5d96612473a22697f: Status 404 returned error can't find the container with id 29b670ca52e3a6622e45bdbdb35563f9d746be9cbd65b6f5d96612473a22697f Jul 07 15:51:59.558180 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:51:59.558136 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-pcsd8" event={"ID":"ed802748-bb8a-410e-bc9b-c342536171c3","Type":"ContainerStarted","Data":"29b670ca52e3a6622e45bdbdb35563f9d746be9cbd65b6f5d96612473a22697f"} Jul 07 15:52:00.566241 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:52:00.566203 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-pcsd8" event={"ID":"ed802748-bb8a-410e-bc9b-c342536171c3","Type":"ContainerStarted","Data":"c37b37321c5c2295be39090313a0bc6d4fdedc83aab917f11ce32f6d3965312e"} Jul 07 15:52:00.590345 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:52:00.590298 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-pcsd8" podStartSLOduration=1.491376356 podStartE2EDuration="2.590281712s" totalImagesPullingTime="1.098905356s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:51:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:52:00.588674 +0000 UTC m=+659.545251489" watchObservedRunningTime="2026-07-07 15:52:00.590281712 +0000 UTC m=+659.546859179" Jul 07 15:52:06.864758 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:52:06.864727 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:52:09.029758 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:52:09.029727 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-pcsd8" Jul 07 15:52:09.030171 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:52:09.029802 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-pcsd8" Jul 07 15:52:09.042359 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:52:09.042332 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-pcsd8" Jul 07 15:52:09.620177 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:52:09.620152 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-pcsd8" Jul 07 15:52:10.796298 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:52:10.796268 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:52:10.902915 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:52:10.902880 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:52:15.366768 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:52:15.366734 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:52:17.284157 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:52:17.284118 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:52:17.421397 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:52:17.421360 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:52:20.106591 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:52:20.106553 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbsqs7b"] Jul 07 15:52:20.110020 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:52:20.109998 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbsqs7b" Jul 07 15:52:20.112781 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:52:20.112760 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Jul 07 15:52:20.127184 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:52:20.127133 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbsqs7b"] Jul 07 15:52:20.144521 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:52:20.144467 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:52:20.171979 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:52:20.171953 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2bc80ed7-bc86-4e0d-8347-4b61a0804f42-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbsqs7b\" (UID: \"2bc80ed7-bc86-4e0d-8347-4b61a0804f42\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbsqs7b" Jul 07 15:52:20.172133 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:52:20.172005 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2bc80ed7-bc86-4e0d-8347-4b61a0804f42-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbsqs7b\" (UID: \"2bc80ed7-bc86-4e0d-8347-4b61a0804f42\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbsqs7b" Jul 07 15:52:20.172133 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:52:20.172023 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2bc80ed7-bc86-4e0d-8347-4b61a0804f42-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbsqs7b\" (UID: \"2bc80ed7-bc86-4e0d-8347-4b61a0804f42\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbsqs7b" Jul 07 15:52:20.172133 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:52:20.172066 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vn48z\" (UniqueName: \"kubernetes.io/projected/2bc80ed7-bc86-4e0d-8347-4b61a0804f42-kube-api-access-vn48z\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbsqs7b\" (UID: \"2bc80ed7-bc86-4e0d-8347-4b61a0804f42\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbsqs7b" Jul 07 15:52:20.172234 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:52:20.172214 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2bc80ed7-bc86-4e0d-8347-4b61a0804f42-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbsqs7b\" (UID: \"2bc80ed7-bc86-4e0d-8347-4b61a0804f42\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbsqs7b" Jul 07 15:52:20.172271 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:52:20.172246 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2bc80ed7-bc86-4e0d-8347-4b61a0804f42-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbsqs7b\" (UID: \"2bc80ed7-bc86-4e0d-8347-4b61a0804f42\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbsqs7b" Jul 07 15:52:20.272771 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:52:20.272736 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2bc80ed7-bc86-4e0d-8347-4b61a0804f42-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbsqs7b\" (UID: \"2bc80ed7-bc86-4e0d-8347-4b61a0804f42\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbsqs7b" Jul 07 15:52:20.272771 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:52:20.272777 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2bc80ed7-bc86-4e0d-8347-4b61a0804f42-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbsqs7b\" (UID: \"2bc80ed7-bc86-4e0d-8347-4b61a0804f42\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbsqs7b" Jul 07 15:52:20.273004 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:52:20.272799 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2bc80ed7-bc86-4e0d-8347-4b61a0804f42-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbsqs7b\" (UID: \"2bc80ed7-bc86-4e0d-8347-4b61a0804f42\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbsqs7b" Jul 07 15:52:20.273004 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:52:20.272845 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2bc80ed7-bc86-4e0d-8347-4b61a0804f42-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbsqs7b\" (UID: \"2bc80ed7-bc86-4e0d-8347-4b61a0804f42\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbsqs7b" Jul 07 15:52:20.273004 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:52:20.272863 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2bc80ed7-bc86-4e0d-8347-4b61a0804f42-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbsqs7b\" (UID: \"2bc80ed7-bc86-4e0d-8347-4b61a0804f42\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbsqs7b" Jul 07 15:52:20.273004 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:52:20.272878 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vn48z\" (UniqueName: \"kubernetes.io/projected/2bc80ed7-bc86-4e0d-8347-4b61a0804f42-kube-api-access-vn48z\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbsqs7b\" (UID: \"2bc80ed7-bc86-4e0d-8347-4b61a0804f42\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbsqs7b" Jul 07 15:52:20.273273 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:52:20.273243 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2bc80ed7-bc86-4e0d-8347-4b61a0804f42-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbsqs7b\" (UID: \"2bc80ed7-bc86-4e0d-8347-4b61a0804f42\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbsqs7b" Jul 07 15:52:20.273327 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:52:20.273270 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2bc80ed7-bc86-4e0d-8347-4b61a0804f42-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbsqs7b\" (UID: \"2bc80ed7-bc86-4e0d-8347-4b61a0804f42\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbsqs7b" Jul 07 15:52:20.274172 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:52:20.274126 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2bc80ed7-bc86-4e0d-8347-4b61a0804f42-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbsqs7b\" (UID: \"2bc80ed7-bc86-4e0d-8347-4b61a0804f42\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbsqs7b" Jul 07 15:52:20.275988 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:52:20.275968 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2bc80ed7-bc86-4e0d-8347-4b61a0804f42-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbsqs7b\" (UID: \"2bc80ed7-bc86-4e0d-8347-4b61a0804f42\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbsqs7b" Jul 07 15:52:20.276094 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:52:20.275985 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2bc80ed7-bc86-4e0d-8347-4b61a0804f42-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbsqs7b\" (UID: \"2bc80ed7-bc86-4e0d-8347-4b61a0804f42\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbsqs7b" Jul 07 15:52:20.284010 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:52:20.283979 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vn48z\" (UniqueName: \"kubernetes.io/projected/2bc80ed7-bc86-4e0d-8347-4b61a0804f42-kube-api-access-vn48z\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbsqs7b\" (UID: \"2bc80ed7-bc86-4e0d-8347-4b61a0804f42\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbsqs7b" Jul 07 15:52:20.420916 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:52:20.420830 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbsqs7b" Jul 07 15:52:20.554450 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:52:20.554428 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbsqs7b"] Jul 07 15:52:20.556629 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:52:20.556602 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bc80ed7_bc86_4e0d_8347_4b61a0804f42.slice/crio-8bff43b923ba6a5f7a0b21aeaf3155abd50fe7a34ad689d76bd8ff216d727c70 WatchSource:0}: Error finding container 8bff43b923ba6a5f7a0b21aeaf3155abd50fe7a34ad689d76bd8ff216d727c70: Status 404 returned error can't find the container with id 8bff43b923ba6a5f7a0b21aeaf3155abd50fe7a34ad689d76bd8ff216d727c70 Jul 07 15:52:20.557969 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:52:20.557954 2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 07 15:52:20.663812 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:52:20.663779 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbsqs7b" event={"ID":"2bc80ed7-bc86-4e0d-8347-4b61a0804f42","Type":"ContainerStarted","Data":"8bff43b923ba6a5f7a0b21aeaf3155abd50fe7a34ad689d76bd8ff216d727c70"} Jul 07 15:52:21.669402 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:52:21.669363 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbsqs7b" event={"ID":"2bc80ed7-bc86-4e0d-8347-4b61a0804f42","Type":"ContainerStarted","Data":"37b7308f5fd9fed9aff59b527047e9428259c6034d4632899f9dfc9ef36f1483"} Jul 07 15:52:21.692524 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:52:21.692461 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbsqs7b" podStartSLOduration=1.4856446810000001 podStartE2EDuration="1.692445846s" totalImagesPullingTime="206.801165ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:52:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:52:21.690509664 +0000 UTC m=+680.647087122" watchObservedRunningTime="2026-07-07 15:52:21.692445846 +0000 UTC m=+680.649023334" Jul 07 15:52:30.421239 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:52:30.421182 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbsqs7b" Jul 07 15:52:30.421821 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:52:30.421273 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbsqs7b" Jul 07 15:52:30.433881 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:52:30.433851 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbsqs7b" Jul 07 15:52:30.723023 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:52:30.722945 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbsqs7b" Jul 07 15:52:37.604926 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:52:37.604895 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:53:10.798399 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:53:10.798327 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:53:39.230896 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:53:39.230864 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:54:13.311117 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:54:13.311076 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-78c657d4d7-lrkgg"] Jul 07 15:54:13.314732 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:54:13.314709 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-78c657d4d7-lrkgg" Jul 07 15:54:13.323183 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:54:13.323157 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-78c657d4d7-lrkgg"] Jul 07 15:54:13.420023 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:54:13.419987 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/cb1758d6-32e9-4d5d-b9f8-7d3b1b16e303-tls-cert\") pod \"authorino-78c657d4d7-lrkgg\" (UID: \"cb1758d6-32e9-4d5d-b9f8-7d3b1b16e303\") " pod="kuadrant-system/authorino-78c657d4d7-lrkgg" Jul 07 15:54:13.420204 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:54:13.420075 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mcnhb\" (UniqueName: \"kubernetes.io/projected/cb1758d6-32e9-4d5d-b9f8-7d3b1b16e303-kube-api-access-mcnhb\") pod \"authorino-78c657d4d7-lrkgg\" (UID: \"cb1758d6-32e9-4d5d-b9f8-7d3b1b16e303\") " pod="kuadrant-system/authorino-78c657d4d7-lrkgg" Jul 07 15:54:13.520763 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:54:13.520723 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/cb1758d6-32e9-4d5d-b9f8-7d3b1b16e303-tls-cert\") pod \"authorino-78c657d4d7-lrkgg\" (UID: \"cb1758d6-32e9-4d5d-b9f8-7d3b1b16e303\") " pod="kuadrant-system/authorino-78c657d4d7-lrkgg" Jul 07 15:54:13.520923 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:54:13.520889 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mcnhb\" (UniqueName: \"kubernetes.io/projected/cb1758d6-32e9-4d5d-b9f8-7d3b1b16e303-kube-api-access-mcnhb\") pod \"authorino-78c657d4d7-lrkgg\" (UID: \"cb1758d6-32e9-4d5d-b9f8-7d3b1b16e303\") " pod="kuadrant-system/authorino-78c657d4d7-lrkgg" Jul 07 15:54:13.523137 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:54:13.523119 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/cb1758d6-32e9-4d5d-b9f8-7d3b1b16e303-tls-cert\") pod \"authorino-78c657d4d7-lrkgg\" (UID: \"cb1758d6-32e9-4d5d-b9f8-7d3b1b16e303\") " pod="kuadrant-system/authorino-78c657d4d7-lrkgg" Jul 07 15:54:13.534123 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:54:13.534101 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mcnhb\" (UniqueName: \"kubernetes.io/projected/cb1758d6-32e9-4d5d-b9f8-7d3b1b16e303-kube-api-access-mcnhb\") pod \"authorino-78c657d4d7-lrkgg\" (UID: \"cb1758d6-32e9-4d5d-b9f8-7d3b1b16e303\") " pod="kuadrant-system/authorino-78c657d4d7-lrkgg" Jul 07 15:54:13.624848 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:54:13.624753 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-78c657d4d7-lrkgg" Jul 07 15:54:13.637777 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:54:13.637747 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:54:13.746265 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:54:13.746232 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:54:13.763904 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:54:13.763879 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-78c657d4d7-lrkgg"] Jul 07 15:54:13.766014 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:54:13.765985 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcb1758d6_32e9_4d5d_b9f8_7d3b1b16e303.slice/crio-a52d2091ecd02b2d0066a58bcd330e45c3b9f58dcbe152c0d0af17eded6ea6ee WatchSource:0}: Error finding container a52d2091ecd02b2d0066a58bcd330e45c3b9f58dcbe152c0d0af17eded6ea6ee: Status 404 returned error can't find the container with id a52d2091ecd02b2d0066a58bcd330e45c3b9f58dcbe152c0d0af17eded6ea6ee Jul 07 15:54:14.213161 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:54:14.213122 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-78c657d4d7-lrkgg" event={"ID":"cb1758d6-32e9-4d5d-b9f8-7d3b1b16e303","Type":"ContainerStarted","Data":"a52d2091ecd02b2d0066a58bcd330e45c3b9f58dcbe152c0d0af17eded6ea6ee"} Jul 07 15:54:14.829742 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:54:14.829695 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:54:14.932368 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:54:14.932265 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:54:15.220572 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:54:15.220410 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-78c657d4d7-lrkgg" event={"ID":"cb1758d6-32e9-4d5d-b9f8-7d3b1b16e303","Type":"ContainerStarted","Data":"2f5371e741eea85631d438e86d370c010d992454377b0ddaf12279b293fdf14e"} Jul 07 15:54:15.246964 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:54:15.246896 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-78c657d4d7-lrkgg" podStartSLOduration=1.8488343619999998 podStartE2EDuration="2.246874125s" totalImagesPullingTime="398.039763ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:54:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:54:15.244715183 +0000 UTC m=+794.201292650" watchObservedRunningTime="2026-07-07 15:54:15.246874125 +0000 UTC m=+794.203451593" Jul 07 15:54:15.294052 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:54:15.294018 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-84bb5794f4-bxncs"] Jul 07 15:54:15.294230 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:54:15.294208 2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/authorino-84bb5794f4-bxncs" podUID="97baad96-0f60-4c9f-84b5-7bd018f22ac9" containerName="authorino" containerID="cri-o://bcfbbdf83b1e3105820e5aa7dbd623d3bc1911baeca6e02511112b97a4852d10" gracePeriod=30 Jul 07 15:54:15.560553 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:54:15.560523 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-84bb5794f4-bxncs" Jul 07 15:54:15.642726 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:54:15.642688 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5f7mp\" (UniqueName: \"kubernetes.io/projected/97baad96-0f60-4c9f-84b5-7bd018f22ac9-kube-api-access-5f7mp\") pod \"97baad96-0f60-4c9f-84b5-7bd018f22ac9\" (UID: \"97baad96-0f60-4c9f-84b5-7bd018f22ac9\") " Jul 07 15:54:15.642918 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:54:15.642771 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/97baad96-0f60-4c9f-84b5-7bd018f22ac9-tls-cert\") pod \"97baad96-0f60-4c9f-84b5-7bd018f22ac9\" (UID: \"97baad96-0f60-4c9f-84b5-7bd018f22ac9\") " Jul 07 15:54:15.644854 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:54:15.644822 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97baad96-0f60-4c9f-84b5-7bd018f22ac9-kube-api-access-5f7mp" (OuterVolumeSpecName: "kube-api-access-5f7mp") pod "97baad96-0f60-4c9f-84b5-7bd018f22ac9" (UID: "97baad96-0f60-4c9f-84b5-7bd018f22ac9"). InnerVolumeSpecName "kube-api-access-5f7mp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 15:54:15.654351 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:54:15.654321 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97baad96-0f60-4c9f-84b5-7bd018f22ac9-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "97baad96-0f60-4c9f-84b5-7bd018f22ac9" (UID: "97baad96-0f60-4c9f-84b5-7bd018f22ac9"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 15:54:15.744416 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:54:15.744373 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5f7mp\" (UniqueName: \"kubernetes.io/projected/97baad96-0f60-4c9f-84b5-7bd018f22ac9-kube-api-access-5f7mp\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 07 15:54:15.744416 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:54:15.744404 2565 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/97baad96-0f60-4c9f-84b5-7bd018f22ac9-tls-cert\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 07 15:54:16.226473 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:54:16.226442 2565 generic.go:356] "Generic (PLEG): container finished" podID="97baad96-0f60-4c9f-84b5-7bd018f22ac9" containerID="bcfbbdf83b1e3105820e5aa7dbd623d3bc1911baeca6e02511112b97a4852d10" exitCode=0 Jul 07 15:54:16.226978 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:54:16.226484 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-84bb5794f4-bxncs" Jul 07 15:54:16.226978 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:54:16.226522 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-84bb5794f4-bxncs" event={"ID":"97baad96-0f60-4c9f-84b5-7bd018f22ac9","Type":"ContainerDied","Data":"bcfbbdf83b1e3105820e5aa7dbd623d3bc1911baeca6e02511112b97a4852d10"} Jul 07 15:54:16.226978 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:54:16.226564 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-84bb5794f4-bxncs" event={"ID":"97baad96-0f60-4c9f-84b5-7bd018f22ac9","Type":"ContainerDied","Data":"2629dfe6e5879696ec4f0c815b6ff32fe559a2b637c8c108b89d1ea6508c8042"} Jul 07 15:54:16.226978 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:54:16.226584 2565 scope.go:117] "RemoveContainer" containerID="bcfbbdf83b1e3105820e5aa7dbd623d3bc1911baeca6e02511112b97a4852d10" Jul 07 15:54:16.236863 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:54:16.236836 2565 scope.go:117] "RemoveContainer" containerID="bcfbbdf83b1e3105820e5aa7dbd623d3bc1911baeca6e02511112b97a4852d10" Jul 07 15:54:16.237169 ip-10-0-132-65 kubenswrapper[2565]: E0707 15:54:16.237147 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bcfbbdf83b1e3105820e5aa7dbd623d3bc1911baeca6e02511112b97a4852d10\": container with ID starting with bcfbbdf83b1e3105820e5aa7dbd623d3bc1911baeca6e02511112b97a4852d10 not found: ID does not exist" containerID="bcfbbdf83b1e3105820e5aa7dbd623d3bc1911baeca6e02511112b97a4852d10" Jul 07 15:54:16.237246 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:54:16.237181 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bcfbbdf83b1e3105820e5aa7dbd623d3bc1911baeca6e02511112b97a4852d10"} err="failed to get container status \"bcfbbdf83b1e3105820e5aa7dbd623d3bc1911baeca6e02511112b97a4852d10\": rpc error: code = NotFound desc = could not find container \"bcfbbdf83b1e3105820e5aa7dbd623d3bc1911baeca6e02511112b97a4852d10\": container with ID starting with bcfbbdf83b1e3105820e5aa7dbd623d3bc1911baeca6e02511112b97a4852d10 not found: ID does not exist" Jul 07 15:54:16.263034 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:54:16.262997 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-84bb5794f4-bxncs"] Jul 07 15:54:16.273413 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:54:16.273383 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-84bb5794f4-bxncs"] Jul 07 15:54:17.672600 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:54:17.672555 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="97baad96-0f60-4c9f-84b5-7bd018f22ac9" path="/var/lib/kubelet/pods/97baad96-0f60-4c9f-84b5-7bd018f22ac9/volumes" Jul 07 15:55:28.550408 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:55:28.550368 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:55:28.610360 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:55:28.610326 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-586d4c4bbb-fpxn2"] Jul 07 15:55:28.610567 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:55:28.610546 2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-controller-586d4c4bbb-fpxn2" podUID="bb501c65-31db-438e-b944-3f553af1b836" containerName="manager" containerID="cri-o://d928d64ce7bdb24de776344f33a710b4ba17f252ae7545855c7dd03092f31102" gracePeriod=30 Jul 07 15:55:28.865857 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:55:28.865833 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-586d4c4bbb-fpxn2" Jul 07 15:55:29.004596 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:55:29.004559 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f6hm4\" (UniqueName: \"kubernetes.io/projected/bb501c65-31db-438e-b944-3f553af1b836-kube-api-access-f6hm4\") pod \"bb501c65-31db-438e-b944-3f553af1b836\" (UID: \"bb501c65-31db-438e-b944-3f553af1b836\") " Jul 07 15:55:29.004797 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:55:29.004638 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/bb501c65-31db-438e-b944-3f553af1b836-webhook-cert\") pod \"bb501c65-31db-438e-b944-3f553af1b836\" (UID: \"bb501c65-31db-438e-b944-3f553af1b836\") " Jul 07 15:55:29.007046 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:55:29.006993 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb501c65-31db-438e-b944-3f553af1b836-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "bb501c65-31db-438e-b944-3f553af1b836" (UID: "bb501c65-31db-438e-b944-3f553af1b836"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 15:55:29.007046 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:55:29.006999 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb501c65-31db-438e-b944-3f553af1b836-kube-api-access-f6hm4" (OuterVolumeSpecName: "kube-api-access-f6hm4") pod "bb501c65-31db-438e-b944-3f553af1b836" (UID: "bb501c65-31db-438e-b944-3f553af1b836"). InnerVolumeSpecName "kube-api-access-f6hm4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 15:55:29.105828 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:55:29.105741 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f6hm4\" (UniqueName: \"kubernetes.io/projected/bb501c65-31db-438e-b944-3f553af1b836-kube-api-access-f6hm4\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 07 15:55:29.105828 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:55:29.105770 2565 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/bb501c65-31db-438e-b944-3f553af1b836-webhook-cert\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 07 15:55:29.568875 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:55:29.568846 2565 generic.go:356] "Generic (PLEG): container finished" podID="bb501c65-31db-438e-b944-3f553af1b836" containerID="d928d64ce7bdb24de776344f33a710b4ba17f252ae7545855c7dd03092f31102" exitCode=0 Jul 07 15:55:29.569296 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:55:29.568891 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-586d4c4bbb-fpxn2" Jul 07 15:55:29.569296 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:55:29.568907 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-586d4c4bbb-fpxn2" event={"ID":"bb501c65-31db-438e-b944-3f553af1b836","Type":"ContainerDied","Data":"d928d64ce7bdb24de776344f33a710b4ba17f252ae7545855c7dd03092f31102"} Jul 07 15:55:29.569296 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:55:29.568948 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-586d4c4bbb-fpxn2" event={"ID":"bb501c65-31db-438e-b944-3f553af1b836","Type":"ContainerDied","Data":"80b7091873a887498f02c25a427f21787edeab9d0c764cf918f5be897522a604"} Jul 07 15:55:29.569296 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:55:29.568969 2565 scope.go:117] "RemoveContainer" containerID="d928d64ce7bdb24de776344f33a710b4ba17f252ae7545855c7dd03092f31102" Jul 07 15:55:29.578386 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:55:29.578368 2565 scope.go:117] "RemoveContainer" containerID="d928d64ce7bdb24de776344f33a710b4ba17f252ae7545855c7dd03092f31102" Jul 07 15:55:29.578713 ip-10-0-132-65 kubenswrapper[2565]: E0707 15:55:29.578694 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d928d64ce7bdb24de776344f33a710b4ba17f252ae7545855c7dd03092f31102\": container with ID starting with d928d64ce7bdb24de776344f33a710b4ba17f252ae7545855c7dd03092f31102 not found: ID does not exist" containerID="d928d64ce7bdb24de776344f33a710b4ba17f252ae7545855c7dd03092f31102" Jul 07 15:55:29.578768 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:55:29.578725 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d928d64ce7bdb24de776344f33a710b4ba17f252ae7545855c7dd03092f31102"} err="failed to get container status \"d928d64ce7bdb24de776344f33a710b4ba17f252ae7545855c7dd03092f31102\": rpc error: code = NotFound desc = could not find container \"d928d64ce7bdb24de776344f33a710b4ba17f252ae7545855c7dd03092f31102\": container with ID starting with d928d64ce7bdb24de776344f33a710b4ba17f252ae7545855c7dd03092f31102 not found: ID does not exist" Jul 07 15:55:29.600502 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:55:29.600464 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-586d4c4bbb-fpxn2"] Jul 07 15:55:29.609591 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:55:29.609567 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-586d4c4bbb-fpxn2"] Jul 07 15:55:29.660672 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:55:29.660642 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bb501c65-31db-438e-b944-3f553af1b836" path="/var/lib/kubelet/pods/bb501c65-31db-438e-b944-3f553af1b836/volumes" Jul 07 15:55:30.323950 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:55:30.323921 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:55:30.423897 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:55:30.423861 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-586d4c4bbb-knwts"] Jul 07 15:55:30.424239 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:55:30.424224 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="97baad96-0f60-4c9f-84b5-7bd018f22ac9" containerName="authorino" Jul 07 15:55:30.424239 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:55:30.424237 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="97baad96-0f60-4c9f-84b5-7bd018f22ac9" containerName="authorino" Jul 07 15:55:30.424336 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:55:30.424248 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb501c65-31db-438e-b944-3f553af1b836" containerName="manager" Jul 07 15:55:30.424336 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:55:30.424254 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb501c65-31db-438e-b944-3f553af1b836" containerName="manager" Jul 07 15:55:30.424336 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:55:30.424314 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="97baad96-0f60-4c9f-84b5-7bd018f22ac9" containerName="authorino" Jul 07 15:55:30.424336 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:55:30.424325 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="bb501c65-31db-438e-b944-3f553af1b836" containerName="manager" Jul 07 15:55:30.429376 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:55:30.429357 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-586d4c4bbb-knwts" Jul 07 15:55:30.434287 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:55:30.434261 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 07 15:55:30.434287 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:55:30.434285 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-rdqhj\"" Jul 07 15:55:30.434456 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:55:30.434297 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 07 15:55:30.440980 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:55:30.440959 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-586d4c4bbb-knwts"] Jul 07 15:55:30.518773 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:55:30.518733 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a399b6c7-1bdf-47da-a5ee-6a3251cfc136-webhook-cert\") pod \"maas-controller-586d4c4bbb-knwts\" (UID: \"a399b6c7-1bdf-47da-a5ee-6a3251cfc136\") " pod="opendatahub/maas-controller-586d4c4bbb-knwts" Jul 07 15:55:30.518972 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:55:30.518805 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvlhg\" (UniqueName: \"kubernetes.io/projected/a399b6c7-1bdf-47da-a5ee-6a3251cfc136-kube-api-access-hvlhg\") pod \"maas-controller-586d4c4bbb-knwts\" (UID: \"a399b6c7-1bdf-47da-a5ee-6a3251cfc136\") " pod="opendatahub/maas-controller-586d4c4bbb-knwts" Jul 07 15:55:30.619430 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:55:30.619332 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a399b6c7-1bdf-47da-a5ee-6a3251cfc136-webhook-cert\") pod \"maas-controller-586d4c4bbb-knwts\" (UID: \"a399b6c7-1bdf-47da-a5ee-6a3251cfc136\") " pod="opendatahub/maas-controller-586d4c4bbb-knwts" Jul 07 15:55:30.619430 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:55:30.619373 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvlhg\" (UniqueName: \"kubernetes.io/projected/a399b6c7-1bdf-47da-a5ee-6a3251cfc136-kube-api-access-hvlhg\") pod \"maas-controller-586d4c4bbb-knwts\" (UID: \"a399b6c7-1bdf-47da-a5ee-6a3251cfc136\") " pod="opendatahub/maas-controller-586d4c4bbb-knwts" Jul 07 15:55:30.621817 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:55:30.621782 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a399b6c7-1bdf-47da-a5ee-6a3251cfc136-webhook-cert\") pod \"maas-controller-586d4c4bbb-knwts\" (UID: \"a399b6c7-1bdf-47da-a5ee-6a3251cfc136\") " pod="opendatahub/maas-controller-586d4c4bbb-knwts" Jul 07 15:55:30.630241 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:55:30.630217 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvlhg\" (UniqueName: \"kubernetes.io/projected/a399b6c7-1bdf-47da-a5ee-6a3251cfc136-kube-api-access-hvlhg\") pod \"maas-controller-586d4c4bbb-knwts\" (UID: \"a399b6c7-1bdf-47da-a5ee-6a3251cfc136\") " pod="opendatahub/maas-controller-586d4c4bbb-knwts" Jul 07 15:55:30.741613 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:55:30.741579 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-586d4c4bbb-knwts" Jul 07 15:55:30.886905 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:55:30.886871 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-586d4c4bbb-knwts"] Jul 07 15:55:30.888228 ip-10-0-132-65 kubenswrapper[2565]: W0707 15:55:30.888200 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda399b6c7_1bdf_47da_a5ee_6a3251cfc136.slice/crio-6c8e200960c805895c4ba84697f87b23110581d8031c08364e8744dcd55946a8 WatchSource:0}: Error finding container 6c8e200960c805895c4ba84697f87b23110581d8031c08364e8744dcd55946a8: Status 404 returned error can't find the container with id 6c8e200960c805895c4ba84697f87b23110581d8031c08364e8744dcd55946a8 Jul 07 15:55:31.581959 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:55:31.581920 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-586d4c4bbb-knwts" event={"ID":"a399b6c7-1bdf-47da-a5ee-6a3251cfc136","Type":"ContainerStarted","Data":"96803365b00cca68e4735dff2c5d3693a4680f7f719a31678751c6e87be9b86e"} Jul 07 15:55:31.582196 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:55:31.582005 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-586d4c4bbb-knwts" event={"ID":"a399b6c7-1bdf-47da-a5ee-6a3251cfc136","Type":"ContainerStarted","Data":"6c8e200960c805895c4ba84697f87b23110581d8031c08364e8744dcd55946a8"} Jul 07 15:55:31.582196 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:55:31.582056 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-586d4c4bbb-knwts" Jul 07 15:55:31.608313 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:55:31.608267 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-586d4c4bbb-knwts" podStartSLOduration=1.207215664 podStartE2EDuration="1.608251116s" totalImagesPullingTime="401.035452ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:55:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:55:31.606864959 +0000 UTC m=+870.563442428" watchObservedRunningTime="2026-07-07 15:55:31.608251116 +0000 UTC m=+870.564828582" Jul 07 15:55:42.594005 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:55:42.593973 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-586d4c4bbb-knwts" Jul 07 15:56:01.536289 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:56:01.536214 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-s9kl8_62da0165-d32c-4bba-9844-a49154d765e5/console-operator/1.log" Jul 07 15:56:01.547108 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:56:01.547074 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-s9kl8_62da0165-d32c-4bba-9844-a49154d765e5/console-operator/1.log" Jul 07 15:56:01.548144 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:56:01.548122 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-942xh_31ff16f0-6054-4eba-942a-b07a5a7afac7/ovn-acl-logging/0.log" Jul 07 15:56:01.552452 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:56:01.552431 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-942xh_31ff16f0-6054-4eba-942a-b07a5a7afac7/ovn-acl-logging/0.log" Jul 07 15:56:09.654835 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:56:09.654799 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:56:09.764065 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:56:09.764034 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:56:09.889751 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:56:09.889713 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:56:16.967184 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:56:16.967151 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:56:17.138119 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:56:17.138086 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:56:17.261393 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:56:17.261304 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:56:17.368677 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:56:17.368648 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:56:24.457858 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:56:24.457822 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:56:24.560538 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:56:24.560507 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:57:46.398285 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:57:46.398204 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:57:50.019565 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:57:50.019520 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:57:50.125846 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:57:50.125810 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 15:58:23.198436 ip-10-0-132-65 kubenswrapper[2565]: I0707 15:58:23.198400 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:00:00.197721 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:00:00.197688 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-key-cleanup-29724000-2m42d"] Jul 07 16:00:00.200931 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:00:00.200914 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29724000-2m42d" Jul 07 16:00:00.209004 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:00:00.208986 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-dockercfg-9bqq6\"" Jul 07 16:00:00.235037 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:00:00.235003 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-key-cleanup-29724000-2m42d"] Jul 07 16:00:00.263292 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:00:00.263263 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ghqj\" (UniqueName: \"kubernetes.io/projected/1669c122-070a-4df8-876d-dac9e83c14f2-kube-api-access-6ghqj\") pod \"maas-api-key-cleanup-29724000-2m42d\" (UID: \"1669c122-070a-4df8-876d-dac9e83c14f2\") " pod="opendatahub/maas-api-key-cleanup-29724000-2m42d" Jul 07 16:00:00.364642 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:00:00.364603 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ghqj\" (UniqueName: \"kubernetes.io/projected/1669c122-070a-4df8-876d-dac9e83c14f2-kube-api-access-6ghqj\") pod \"maas-api-key-cleanup-29724000-2m42d\" (UID: \"1669c122-070a-4df8-876d-dac9e83c14f2\") " pod="opendatahub/maas-api-key-cleanup-29724000-2m42d" Jul 07 16:00:00.380187 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:00:00.380160 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ghqj\" (UniqueName: \"kubernetes.io/projected/1669c122-070a-4df8-876d-dac9e83c14f2-kube-api-access-6ghqj\") pod \"maas-api-key-cleanup-29724000-2m42d\" (UID: \"1669c122-070a-4df8-876d-dac9e83c14f2\") " pod="opendatahub/maas-api-key-cleanup-29724000-2m42d" Jul 07 16:00:00.511525 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:00:00.511417 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29724000-2m42d" Jul 07 16:00:00.646646 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:00:00.646621 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-key-cleanup-29724000-2m42d"] Jul 07 16:00:00.648858 ip-10-0-132-65 kubenswrapper[2565]: W0707 16:00:00.648826 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1669c122_070a_4df8_876d_dac9e83c14f2.slice/crio-2bedf8fc953ed8d580adf887cc473ab65130c7fdb62266101c1d6b64b48dfb6f WatchSource:0}: Error finding container 2bedf8fc953ed8d580adf887cc473ab65130c7fdb62266101c1d6b64b48dfb6f: Status 404 returned error can't find the container with id 2bedf8fc953ed8d580adf887cc473ab65130c7fdb62266101c1d6b64b48dfb6f Jul 07 16:00:00.650707 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:00:00.650688 2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 07 16:00:00.876610 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:00:00.876571 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29724000-2m42d" event={"ID":"1669c122-070a-4df8-876d-dac9e83c14f2","Type":"ContainerStarted","Data":"2bedf8fc953ed8d580adf887cc473ab65130c7fdb62266101c1d6b64b48dfb6f"} Jul 07 16:00:05.904477 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:00:05.904389 2565 generic.go:356] "Generic (PLEG): container finished" podID="1669c122-070a-4df8-876d-dac9e83c14f2" containerID="3460d727d553346d0334d36778899580f26a8db49d645ac21ca4a6894b43eab3" exitCode=0 Jul 07 16:00:05.904873 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:00:05.904477 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29724000-2m42d" event={"ID":"1669c122-070a-4df8-876d-dac9e83c14f2","Type":"ContainerDied","Data":"3460d727d553346d0334d36778899580f26a8db49d645ac21ca4a6894b43eab3"} Jul 07 16:00:07.051036 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:00:07.051004 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29724000-2m42d" Jul 07 16:00:07.128644 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:00:07.128593 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6ghqj\" (UniqueName: \"kubernetes.io/projected/1669c122-070a-4df8-876d-dac9e83c14f2-kube-api-access-6ghqj\") pod \"1669c122-070a-4df8-876d-dac9e83c14f2\" (UID: \"1669c122-070a-4df8-876d-dac9e83c14f2\") " Jul 07 16:00:07.130725 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:00:07.130690 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1669c122-070a-4df8-876d-dac9e83c14f2-kube-api-access-6ghqj" (OuterVolumeSpecName: "kube-api-access-6ghqj") pod "1669c122-070a-4df8-876d-dac9e83c14f2" (UID: "1669c122-070a-4df8-876d-dac9e83c14f2"). InnerVolumeSpecName "kube-api-access-6ghqj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 16:00:07.230016 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:00:07.229919 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6ghqj\" (UniqueName: \"kubernetes.io/projected/1669c122-070a-4df8-876d-dac9e83c14f2-kube-api-access-6ghqj\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 07 16:00:07.918322 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:00:07.918241 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29724000-2m42d" event={"ID":"1669c122-070a-4df8-876d-dac9e83c14f2","Type":"ContainerDied","Data":"2bedf8fc953ed8d580adf887cc473ab65130c7fdb62266101c1d6b64b48dfb6f"} Jul 07 16:00:07.918322 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:00:07.918321 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2bedf8fc953ed8d580adf887cc473ab65130c7fdb62266101c1d6b64b48dfb6f" Jul 07 16:00:07.918525 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:00:07.918257 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29724000-2m42d" Jul 07 16:00:20.393621 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:00:20.393586 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:00:20.459225 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:00:20.459172 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:01:01.575031 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:01:01.574951 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-s9kl8_62da0165-d32c-4bba-9844-a49154d765e5/console-operator/1.log" Jul 07 16:01:01.576104 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:01:01.576085 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-s9kl8_62da0165-d32c-4bba-9844-a49154d765e5/console-operator/1.log" Jul 07 16:01:01.579144 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:01:01.579130 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-942xh_31ff16f0-6054-4eba-942a-b07a5a7afac7/ovn-acl-logging/0.log" Jul 07 16:01:01.580253 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:01:01.580237 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-942xh_31ff16f0-6054-4eba-942a-b07a5a7afac7/ovn-acl-logging/0.log" Jul 07 16:03:14.474280 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:03:14.474250 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:03:15.996564 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:03:15.996526 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:06:01.606376 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:06:01.606338 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-s9kl8_62da0165-d32c-4bba-9844-a49154d765e5/console-operator/1.log" Jul 07 16:06:01.607335 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:06:01.607303 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-s9kl8_62da0165-d32c-4bba-9844-a49154d765e5/console-operator/1.log" Jul 07 16:06:01.610615 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:06:01.610596 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-942xh_31ff16f0-6054-4eba-942a-b07a5a7afac7/ovn-acl-logging/0.log" Jul 07 16:06:01.611409 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:06:01.611390 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-942xh_31ff16f0-6054-4eba-942a-b07a5a7afac7/ovn-acl-logging/0.log" Jul 07 16:06:02.490702 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:06:02.490653 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:06:22.886831 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:06:22.886797 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:07:16.831973 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:07:16.831918 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:07:26.747946 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:07:26.747910 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:07:35.022931 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:07:35.022896 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:07:40.134538 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:07:40.134468 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:07:43.197693 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:07:43.197644 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:07:43.297309 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:07:43.297276 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:07:43.720560 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:07:43.720520 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:07:50.910258 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:07:50.910229 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:07:50.980538 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:07:50.980483 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:07:51.160698 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:07:51.160614 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-6eebde10-openshift-default-54868ffd59-5b2kt"] Jul 07 16:07:51.161111 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:07:51.161092 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1669c122-070a-4df8-876d-dac9e83c14f2" containerName="cleanup" Jul 07 16:07:51.161227 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:07:51.161115 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="1669c122-070a-4df8-876d-dac9e83c14f2" containerName="cleanup" Jul 07 16:07:51.161227 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:07:51.161207 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="1669c122-070a-4df8-876d-dac9e83c14f2" containerName="cleanup" Jul 07 16:07:51.163272 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:07:51.163251 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-6eebde10-openshift-default-54868ffd59-5b2kt" Jul 07 16:07:51.168613 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:07:51.168591 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-6eebde10-openshift-default-dockercfg-vcjzf\"" Jul 07 16:07:51.178928 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:07:51.178906 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:07:51.230836 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:07:51.230789 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-6eebde10-openshift-default-54868ffd59-5b2kt"] Jul 07 16:07:51.264857 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:07:51.264825 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dc3c3ea-a70d-4ec9-bf5a-83613d8b45a8-workload-socket\") pod \"e2e-ait-6eebde10-openshift-default-54868ffd59-5b2kt\" (UID: \"6dc3c3ea-a70d-4ec9-bf5a-83613d8b45a8\") " pod="openshift-ingress/e2e-ait-6eebde10-openshift-default-54868ffd59-5b2kt" Jul 07 16:07:51.264857 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:07:51.264866 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6dc3c3ea-a70d-4ec9-bf5a-83613d8b45a8-istio-token\") pod \"e2e-ait-6eebde10-openshift-default-54868ffd59-5b2kt\" (UID: \"6dc3c3ea-a70d-4ec9-bf5a-83613d8b45a8\") " pod="openshift-ingress/e2e-ait-6eebde10-openshift-default-54868ffd59-5b2kt" Jul 07 16:07:51.265096 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:07:51.264937 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6dc3c3ea-a70d-4ec9-bf5a-83613d8b45a8-istio-envoy\") pod \"e2e-ait-6eebde10-openshift-default-54868ffd59-5b2kt\" (UID: \"6dc3c3ea-a70d-4ec9-bf5a-83613d8b45a8\") " pod="openshift-ingress/e2e-ait-6eebde10-openshift-default-54868ffd59-5b2kt" Jul 07 16:07:51.265096 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:07:51.264966 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6dc3c3ea-a70d-4ec9-bf5a-83613d8b45a8-istiod-ca-cert\") pod \"e2e-ait-6eebde10-openshift-default-54868ffd59-5b2kt\" (UID: \"6dc3c3ea-a70d-4ec9-bf5a-83613d8b45a8\") " pod="openshift-ingress/e2e-ait-6eebde10-openshift-default-54868ffd59-5b2kt" Jul 07 16:07:51.265096 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:07:51.265054 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6dc3c3ea-a70d-4ec9-bf5a-83613d8b45a8-workload-certs\") pod \"e2e-ait-6eebde10-openshift-default-54868ffd59-5b2kt\" (UID: \"6dc3c3ea-a70d-4ec9-bf5a-83613d8b45a8\") " pod="openshift-ingress/e2e-ait-6eebde10-openshift-default-54868ffd59-5b2kt" Jul 07 16:07:51.265096 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:07:51.265075 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57whh\" (UniqueName: \"kubernetes.io/projected/6dc3c3ea-a70d-4ec9-bf5a-83613d8b45a8-kube-api-access-57whh\") pod \"e2e-ait-6eebde10-openshift-default-54868ffd59-5b2kt\" (UID: \"6dc3c3ea-a70d-4ec9-bf5a-83613d8b45a8\") " pod="openshift-ingress/e2e-ait-6eebde10-openshift-default-54868ffd59-5b2kt" Jul 07 16:07:51.265334 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:07:51.265104 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dc3c3ea-a70d-4ec9-bf5a-83613d8b45a8-credential-socket\") pod \"e2e-ait-6eebde10-openshift-default-54868ffd59-5b2kt\" (UID: \"6dc3c3ea-a70d-4ec9-bf5a-83613d8b45a8\") " pod="openshift-ingress/e2e-ait-6eebde10-openshift-default-54868ffd59-5b2kt" Jul 07 16:07:51.265334 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:07:51.265183 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6dc3c3ea-a70d-4ec9-bf5a-83613d8b45a8-istio-data\") pod \"e2e-ait-6eebde10-openshift-default-54868ffd59-5b2kt\" (UID: \"6dc3c3ea-a70d-4ec9-bf5a-83613d8b45a8\") " pod="openshift-ingress/e2e-ait-6eebde10-openshift-default-54868ffd59-5b2kt" Jul 07 16:07:51.265334 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:07:51.265216 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6dc3c3ea-a70d-4ec9-bf5a-83613d8b45a8-istio-podinfo\") pod \"e2e-ait-6eebde10-openshift-default-54868ffd59-5b2kt\" (UID: \"6dc3c3ea-a70d-4ec9-bf5a-83613d8b45a8\") " pod="openshift-ingress/e2e-ait-6eebde10-openshift-default-54868ffd59-5b2kt" Jul 07 16:07:51.291585 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:07:51.291547 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:07:51.365733 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:07:51.365694 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6dc3c3ea-a70d-4ec9-bf5a-83613d8b45a8-workload-certs\") pod \"e2e-ait-6eebde10-openshift-default-54868ffd59-5b2kt\" (UID: \"6dc3c3ea-a70d-4ec9-bf5a-83613d8b45a8\") " pod="openshift-ingress/e2e-ait-6eebde10-openshift-default-54868ffd59-5b2kt" Jul 07 16:07:51.365733 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:07:51.365735 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57whh\" (UniqueName: \"kubernetes.io/projected/6dc3c3ea-a70d-4ec9-bf5a-83613d8b45a8-kube-api-access-57whh\") pod \"e2e-ait-6eebde10-openshift-default-54868ffd59-5b2kt\" (UID: \"6dc3c3ea-a70d-4ec9-bf5a-83613d8b45a8\") " pod="openshift-ingress/e2e-ait-6eebde10-openshift-default-54868ffd59-5b2kt" Jul 07 16:07:51.365990 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:07:51.365763 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dc3c3ea-a70d-4ec9-bf5a-83613d8b45a8-credential-socket\") pod \"e2e-ait-6eebde10-openshift-default-54868ffd59-5b2kt\" (UID: \"6dc3c3ea-a70d-4ec9-bf5a-83613d8b45a8\") " pod="openshift-ingress/e2e-ait-6eebde10-openshift-default-54868ffd59-5b2kt" Jul 07 16:07:51.365990 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:07:51.365818 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6dc3c3ea-a70d-4ec9-bf5a-83613d8b45a8-istio-data\") pod \"e2e-ait-6eebde10-openshift-default-54868ffd59-5b2kt\" (UID: \"6dc3c3ea-a70d-4ec9-bf5a-83613d8b45a8\") " pod="openshift-ingress/e2e-ait-6eebde10-openshift-default-54868ffd59-5b2kt" Jul 07 16:07:51.365990 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:07:51.365857 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6dc3c3ea-a70d-4ec9-bf5a-83613d8b45a8-istio-podinfo\") pod \"e2e-ait-6eebde10-openshift-default-54868ffd59-5b2kt\" (UID: \"6dc3c3ea-a70d-4ec9-bf5a-83613d8b45a8\") " pod="openshift-ingress/e2e-ait-6eebde10-openshift-default-54868ffd59-5b2kt" Jul 07 16:07:51.365990 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:07:51.365881 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dc3c3ea-a70d-4ec9-bf5a-83613d8b45a8-workload-socket\") pod \"e2e-ait-6eebde10-openshift-default-54868ffd59-5b2kt\" (UID: \"6dc3c3ea-a70d-4ec9-bf5a-83613d8b45a8\") " pod="openshift-ingress/e2e-ait-6eebde10-openshift-default-54868ffd59-5b2kt" Jul 07 16:07:51.365990 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:07:51.365903 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6dc3c3ea-a70d-4ec9-bf5a-83613d8b45a8-istio-token\") pod \"e2e-ait-6eebde10-openshift-default-54868ffd59-5b2kt\" (UID: \"6dc3c3ea-a70d-4ec9-bf5a-83613d8b45a8\") " pod="openshift-ingress/e2e-ait-6eebde10-openshift-default-54868ffd59-5b2kt" Jul 07 16:07:51.365990 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:07:51.365928 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6dc3c3ea-a70d-4ec9-bf5a-83613d8b45a8-istio-envoy\") pod \"e2e-ait-6eebde10-openshift-default-54868ffd59-5b2kt\" (UID: \"6dc3c3ea-a70d-4ec9-bf5a-83613d8b45a8\") " pod="openshift-ingress/e2e-ait-6eebde10-openshift-default-54868ffd59-5b2kt" Jul 07 16:07:51.365990 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:07:51.365954 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6dc3c3ea-a70d-4ec9-bf5a-83613d8b45a8-istiod-ca-cert\") pod \"e2e-ait-6eebde10-openshift-default-54868ffd59-5b2kt\" (UID: \"6dc3c3ea-a70d-4ec9-bf5a-83613d8b45a8\") " pod="openshift-ingress/e2e-ait-6eebde10-openshift-default-54868ffd59-5b2kt" Jul 07 16:07:51.366350 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:07:51.366182 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6dc3c3ea-a70d-4ec9-bf5a-83613d8b45a8-workload-certs\") pod \"e2e-ait-6eebde10-openshift-default-54868ffd59-5b2kt\" (UID: \"6dc3c3ea-a70d-4ec9-bf5a-83613d8b45a8\") " pod="openshift-ingress/e2e-ait-6eebde10-openshift-default-54868ffd59-5b2kt" Jul 07 16:07:51.366350 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:07:51.366224 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dc3c3ea-a70d-4ec9-bf5a-83613d8b45a8-credential-socket\") pod \"e2e-ait-6eebde10-openshift-default-54868ffd59-5b2kt\" (UID: \"6dc3c3ea-a70d-4ec9-bf5a-83613d8b45a8\") " pod="openshift-ingress/e2e-ait-6eebde10-openshift-default-54868ffd59-5b2kt" Jul 07 16:07:51.366477 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:07:51.366447 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dc3c3ea-a70d-4ec9-bf5a-83613d8b45a8-workload-socket\") pod \"e2e-ait-6eebde10-openshift-default-54868ffd59-5b2kt\" (UID: \"6dc3c3ea-a70d-4ec9-bf5a-83613d8b45a8\") " pod="openshift-ingress/e2e-ait-6eebde10-openshift-default-54868ffd59-5b2kt" Jul 07 16:07:51.366551 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:07:51.366506 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6dc3c3ea-a70d-4ec9-bf5a-83613d8b45a8-istio-data\") pod \"e2e-ait-6eebde10-openshift-default-54868ffd59-5b2kt\" (UID: \"6dc3c3ea-a70d-4ec9-bf5a-83613d8b45a8\") " pod="openshift-ingress/e2e-ait-6eebde10-openshift-default-54868ffd59-5b2kt" Jul 07 16:07:51.366739 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:07:51.366722 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6dc3c3ea-a70d-4ec9-bf5a-83613d8b45a8-istiod-ca-cert\") pod \"e2e-ait-6eebde10-openshift-default-54868ffd59-5b2kt\" (UID: \"6dc3c3ea-a70d-4ec9-bf5a-83613d8b45a8\") " pod="openshift-ingress/e2e-ait-6eebde10-openshift-default-54868ffd59-5b2kt" Jul 07 16:07:51.368438 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:07:51.368420 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6dc3c3ea-a70d-4ec9-bf5a-83613d8b45a8-istio-envoy\") pod \"e2e-ait-6eebde10-openshift-default-54868ffd59-5b2kt\" (UID: \"6dc3c3ea-a70d-4ec9-bf5a-83613d8b45a8\") " pod="openshift-ingress/e2e-ait-6eebde10-openshift-default-54868ffd59-5b2kt" Jul 07 16:07:51.368486 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:07:51.368470 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6dc3c3ea-a70d-4ec9-bf5a-83613d8b45a8-istio-podinfo\") pod \"e2e-ait-6eebde10-openshift-default-54868ffd59-5b2kt\" (UID: \"6dc3c3ea-a70d-4ec9-bf5a-83613d8b45a8\") " pod="openshift-ingress/e2e-ait-6eebde10-openshift-default-54868ffd59-5b2kt" Jul 07 16:07:51.377755 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:07:51.377734 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-57whh\" (UniqueName: \"kubernetes.io/projected/6dc3c3ea-a70d-4ec9-bf5a-83613d8b45a8-kube-api-access-57whh\") pod \"e2e-ait-6eebde10-openshift-default-54868ffd59-5b2kt\" (UID: \"6dc3c3ea-a70d-4ec9-bf5a-83613d8b45a8\") " pod="openshift-ingress/e2e-ait-6eebde10-openshift-default-54868ffd59-5b2kt" Jul 07 16:07:51.380686 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:07:51.380662 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6dc3c3ea-a70d-4ec9-bf5a-83613d8b45a8-istio-token\") pod \"e2e-ait-6eebde10-openshift-default-54868ffd59-5b2kt\" (UID: \"6dc3c3ea-a70d-4ec9-bf5a-83613d8b45a8\") " pod="openshift-ingress/e2e-ait-6eebde10-openshift-default-54868ffd59-5b2kt" Jul 07 16:07:51.473372 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:07:51.473275 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-6eebde10-openshift-default-54868ffd59-5b2kt" Jul 07 16:07:51.627631 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:07:51.627598 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-6eebde10-openshift-default-54868ffd59-5b2kt"] Jul 07 16:07:51.631542 ip-10-0-132-65 kubenswrapper[2565]: W0707 16:07:51.631514 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6dc3c3ea_a70d_4ec9_bf5a_83613d8b45a8.slice/crio-a9f36d995d22c9e905b3f01f1b81722e4adb047855e5f7975f287fe154197b77 WatchSource:0}: Error finding container a9f36d995d22c9e905b3f01f1b81722e4adb047855e5f7975f287fe154197b77: Status 404 returned error can't find the container with id a9f36d995d22c9e905b3f01f1b81722e4adb047855e5f7975f287fe154197b77 Jul 07 16:07:51.633579 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:07:51.633550 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 07 16:07:51.633659 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:07:51.633611 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 07 16:07:51.633659 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:07:51.633637 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 07 16:07:52.165897 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:07:52.165864 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-6eebde10-openshift-default-54868ffd59-5b2kt" event={"ID":"6dc3c3ea-a70d-4ec9-bf5a-83613d8b45a8","Type":"ContainerStarted","Data":"8f6f81623d9cb0ca963e55744731f4e2ba52091b80966f4d4758228ee5fc1a80"} Jul 07 16:07:52.166296 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:07:52.165918 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-6eebde10-openshift-default-54868ffd59-5b2kt" event={"ID":"6dc3c3ea-a70d-4ec9-bf5a-83613d8b45a8","Type":"ContainerStarted","Data":"a9f36d995d22c9e905b3f01f1b81722e4adb047855e5f7975f287fe154197b77"} Jul 07 16:07:52.196025 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:07:52.195971 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-6eebde10-openshift-default-54868ffd59-5b2kt" podStartSLOduration=1.195955132 podStartE2EDuration="1.195955132s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:07:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:07:52.194366496 +0000 UTC m=+1611.150943963" watchObservedRunningTime="2026-07-07 16:07:52.195955132 +0000 UTC m=+1611.152532599" Jul 07 16:07:52.367636 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:07:52.367604 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:07:52.468252 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:07:52.468178 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-ait-6eebde10-64d664cd44-4t744"] Jul 07 16:07:52.470807 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:07:52.470792 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-6eebde10-64d664cd44-4t744" Jul 07 16:07:52.473060 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:07:52.473039 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:07:52.474029 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:07:52.474013 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-6eebde10-openshift-default-54868ffd59-5b2kt" Jul 07 16:07:52.475565 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:07:52.475530 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-ait-6eebde10-serving-cert\"" Jul 07 16:07:52.475660 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:07:52.475573 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-dockercfg-9bqq6\"" Jul 07 16:07:52.480040 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:07:52.480017 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-6eebde10-openshift-default-54868ffd59-5b2kt" Jul 07 16:07:52.483367 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:07:52.483345 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-ait-6eebde10-64d664cd44-4t744"] Jul 07 16:07:52.548095 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:07:52.548062 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:07:52.577647 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:07:52.577609 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d65757f-db0f-4b2a-893e-279002e8452a-openshift-service-ca\") pod \"maas-api-e2e-ait-6eebde10-64d664cd44-4t744\" (UID: \"8d65757f-db0f-4b2a-893e-279002e8452a\") " pod="opendatahub/maas-api-e2e-ait-6eebde10-64d664cd44-4t744" Jul 07 16:07:52.577924 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:07:52.577897 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g7jrd\" (UniqueName: \"kubernetes.io/projected/8d65757f-db0f-4b2a-893e-279002e8452a-kube-api-access-g7jrd\") pod \"maas-api-e2e-ait-6eebde10-64d664cd44-4t744\" (UID: \"8d65757f-db0f-4b2a-893e-279002e8452a\") " pod="opendatahub/maas-api-e2e-ait-6eebde10-64d664cd44-4t744" Jul 07 16:07:52.578100 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:07:52.578074 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8d65757f-db0f-4b2a-893e-279002e8452a-maas-api-tls\") pod \"maas-api-e2e-ait-6eebde10-64d664cd44-4t744\" (UID: \"8d65757f-db0f-4b2a-893e-279002e8452a\") " pod="opendatahub/maas-api-e2e-ait-6eebde10-64d664cd44-4t744" Jul 07 16:07:52.636074 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:07:52.636041 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:07:52.678909 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:07:52.678876 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8d65757f-db0f-4b2a-893e-279002e8452a-maas-api-tls\") pod \"maas-api-e2e-ait-6eebde10-64d664cd44-4t744\" (UID: \"8d65757f-db0f-4b2a-893e-279002e8452a\") " pod="opendatahub/maas-api-e2e-ait-6eebde10-64d664cd44-4t744" Jul 07 16:07:52.679090 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:07:52.679035 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d65757f-db0f-4b2a-893e-279002e8452a-openshift-service-ca\") pod \"maas-api-e2e-ait-6eebde10-64d664cd44-4t744\" (UID: \"8d65757f-db0f-4b2a-893e-279002e8452a\") " pod="opendatahub/maas-api-e2e-ait-6eebde10-64d664cd44-4t744" Jul 07 16:07:52.679090 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:07:52.679072 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7jrd\" (UniqueName: \"kubernetes.io/projected/8d65757f-db0f-4b2a-893e-279002e8452a-kube-api-access-g7jrd\") pod \"maas-api-e2e-ait-6eebde10-64d664cd44-4t744\" (UID: \"8d65757f-db0f-4b2a-893e-279002e8452a\") " pod="opendatahub/maas-api-e2e-ait-6eebde10-64d664cd44-4t744" Jul 07 16:07:52.679692 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:07:52.679669 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d65757f-db0f-4b2a-893e-279002e8452a-openshift-service-ca\") pod \"maas-api-e2e-ait-6eebde10-64d664cd44-4t744\" (UID: \"8d65757f-db0f-4b2a-893e-279002e8452a\") " pod="opendatahub/maas-api-e2e-ait-6eebde10-64d664cd44-4t744" Jul 07 16:07:52.681530 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:07:52.681504 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8d65757f-db0f-4b2a-893e-279002e8452a-maas-api-tls\") pod \"maas-api-e2e-ait-6eebde10-64d664cd44-4t744\" (UID: \"8d65757f-db0f-4b2a-893e-279002e8452a\") " pod="opendatahub/maas-api-e2e-ait-6eebde10-64d664cd44-4t744" Jul 07 16:07:52.690836 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:07:52.690819 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7jrd\" (UniqueName: \"kubernetes.io/projected/8d65757f-db0f-4b2a-893e-279002e8452a-kube-api-access-g7jrd\") pod \"maas-api-e2e-ait-6eebde10-64d664cd44-4t744\" (UID: \"8d65757f-db0f-4b2a-893e-279002e8452a\") " pod="opendatahub/maas-api-e2e-ait-6eebde10-64d664cd44-4t744" Jul 07 16:07:52.782732 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:07:52.782656 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-6eebde10-64d664cd44-4t744" Jul 07 16:07:52.927157 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:07:52.927133 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-ait-6eebde10-64d664cd44-4t744"] Jul 07 16:07:52.928826 ip-10-0-132-65 kubenswrapper[2565]: W0707 16:07:52.928800 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d65757f_db0f_4b2a_893e_279002e8452a.slice/crio-1549013f9b4b9da04d08a64347f23f3fef68618980f766fb8f95eae65c939c8e WatchSource:0}: Error finding container 1549013f9b4b9da04d08a64347f23f3fef68618980f766fb8f95eae65c939c8e: Status 404 returned error can't find the container with id 1549013f9b4b9da04d08a64347f23f3fef68618980f766fb8f95eae65c939c8e Jul 07 16:07:52.930196 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:07:52.930179 2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 07 16:07:53.171436 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:07:53.171398 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-6eebde10-64d664cd44-4t744" event={"ID":"8d65757f-db0f-4b2a-893e-279002e8452a","Type":"ContainerStarted","Data":"1549013f9b4b9da04d08a64347f23f3fef68618980f766fb8f95eae65c939c8e"} Jul 07 16:07:53.171858 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:07:53.171707 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-6eebde10-openshift-default-54868ffd59-5b2kt" Jul 07 16:07:53.172806 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:07:53.172784 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-6eebde10-openshift-default-54868ffd59-5b2kt" Jul 07 16:07:53.329964 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:07:53.329930 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:07:53.435336 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:07:53.435238 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:07:54.178109 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:07:54.178070 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-6eebde10-64d664cd44-4t744" event={"ID":"8d65757f-db0f-4b2a-893e-279002e8452a","Type":"ContainerStarted","Data":"a044d61c83513a339f04812741eddadaa69c2b5cba5bb9cfc9da626db038b0d3"} Jul 07 16:07:54.178613 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:07:54.178228 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-ait-6eebde10-64d664cd44-4t744" Jul 07 16:07:54.214372 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:07:54.214318 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-ait-6eebde10-64d664cd44-4t744" podStartSLOduration=1.618467435 podStartE2EDuration="2.214301802s" totalImagesPullingTime="595.834367ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:07:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:07:54.212832905 +0000 UTC m=+1613.169410372" watchObservedRunningTime="2026-07-07 16:07:54.214301802 +0000 UTC m=+1613.170879269" Jul 07 16:07:55.546283 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:07:55.546253 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:07:57.507165 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:07:57.507108 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:07:57.676543 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:07:57.676483 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-6eebde10-openshift-default-54868ffd59-5b2kt"] Jul 07 16:07:57.676777 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:07:57.676723 2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-6eebde10-openshift-default-54868ffd59-5b2kt" podUID="6dc3c3ea-a70d-4ec9-bf5a-83613d8b45a8" containerName="istio-proxy" containerID="cri-o://8f6f81623d9cb0ca963e55744731f4e2ba52091b80966f4d4758228ee5fc1a80" gracePeriod=30 Jul 07 16:07:57.696664 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:07:57.696614 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:07:57.818844 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:07:57.818810 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:07:58.084614 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:07:58.081518 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-ait-6eebde10-64d664cd44-4t744"] Jul 07 16:07:58.084614 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:07:58.081792 2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-ait-6eebde10-64d664cd44-4t744" podUID="8d65757f-db0f-4b2a-893e-279002e8452a" containerName="maas-api" containerID="cri-o://a044d61c83513a339f04812741eddadaa69c2b5cba5bb9cfc9da626db038b0d3" gracePeriod=30 Jul 07 16:07:58.095816 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:07:58.095786 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-ait-6eebde10-64d664cd44-4t744" Jul 07 16:07:58.114706 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:07:58.114682 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:07:58.346268 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:07:58.346199 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-6eebde10-64d664cd44-4t744" Jul 07 16:07:58.440789 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:07:58.440757 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g7jrd\" (UniqueName: \"kubernetes.io/projected/8d65757f-db0f-4b2a-893e-279002e8452a-kube-api-access-g7jrd\") pod \"8d65757f-db0f-4b2a-893e-279002e8452a\" (UID: \"8d65757f-db0f-4b2a-893e-279002e8452a\") " Jul 07 16:07:58.440965 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:07:58.440808 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8d65757f-db0f-4b2a-893e-279002e8452a-maas-api-tls\") pod \"8d65757f-db0f-4b2a-893e-279002e8452a\" (UID: \"8d65757f-db0f-4b2a-893e-279002e8452a\") " Jul 07 16:07:58.440965 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:07:58.440892 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d65757f-db0f-4b2a-893e-279002e8452a-openshift-service-ca\") pod \"8d65757f-db0f-4b2a-893e-279002e8452a\" (UID: \"8d65757f-db0f-4b2a-893e-279002e8452a\") " Jul 07 16:07:58.441263 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:07:58.441235 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d65757f-db0f-4b2a-893e-279002e8452a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "8d65757f-db0f-4b2a-893e-279002e8452a" (UID: "8d65757f-db0f-4b2a-893e-279002e8452a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 16:07:58.442899 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:07:58.442877 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d65757f-db0f-4b2a-893e-279002e8452a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "8d65757f-db0f-4b2a-893e-279002e8452a" (UID: "8d65757f-db0f-4b2a-893e-279002e8452a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 16:07:58.442998 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:07:58.442895 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d65757f-db0f-4b2a-893e-279002e8452a-kube-api-access-g7jrd" (OuterVolumeSpecName: "kube-api-access-g7jrd") pod "8d65757f-db0f-4b2a-893e-279002e8452a" (UID: "8d65757f-db0f-4b2a-893e-279002e8452a"). InnerVolumeSpecName "kube-api-access-g7jrd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 16:07:58.542016 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:07:58.541974 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8d65757f-db0f-4b2a-893e-279002e8452a-maas-api-tls\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 07 16:07:58.542016 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:07:58.542009 2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d65757f-db0f-4b2a-893e-279002e8452a-openshift-service-ca\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 07 16:07:58.542016 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:07:58.542020 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g7jrd\" (UniqueName: \"kubernetes.io/projected/8d65757f-db0f-4b2a-893e-279002e8452a-kube-api-access-g7jrd\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 07 16:07:59.204352 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:07:59.204316 2565 generic.go:356] "Generic (PLEG): container finished" podID="8d65757f-db0f-4b2a-893e-279002e8452a" containerID="a044d61c83513a339f04812741eddadaa69c2b5cba5bb9cfc9da626db038b0d3" exitCode=0 Jul 07 16:07:59.204589 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:07:59.204360 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-6eebde10-64d664cd44-4t744" Jul 07 16:07:59.204589 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:07:59.204396 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-6eebde10-64d664cd44-4t744" event={"ID":"8d65757f-db0f-4b2a-893e-279002e8452a","Type":"ContainerDied","Data":"a044d61c83513a339f04812741eddadaa69c2b5cba5bb9cfc9da626db038b0d3"} Jul 07 16:07:59.204589 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:07:59.204447 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-6eebde10-64d664cd44-4t744" event={"ID":"8d65757f-db0f-4b2a-893e-279002e8452a","Type":"ContainerDied","Data":"1549013f9b4b9da04d08a64347f23f3fef68618980f766fb8f95eae65c939c8e"} Jul 07 16:07:59.204589 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:07:59.204469 2565 scope.go:117] "RemoveContainer" containerID="a044d61c83513a339f04812741eddadaa69c2b5cba5bb9cfc9da626db038b0d3" Jul 07 16:07:59.215745 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:07:59.214111 2565 scope.go:117] "RemoveContainer" containerID="a044d61c83513a339f04812741eddadaa69c2b5cba5bb9cfc9da626db038b0d3" Jul 07 16:07:59.215974 ip-10-0-132-65 kubenswrapper[2565]: E0707 16:07:59.215957 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a044d61c83513a339f04812741eddadaa69c2b5cba5bb9cfc9da626db038b0d3\": container with ID starting with a044d61c83513a339f04812741eddadaa69c2b5cba5bb9cfc9da626db038b0d3 not found: ID does not exist" containerID="a044d61c83513a339f04812741eddadaa69c2b5cba5bb9cfc9da626db038b0d3" Jul 07 16:07:59.216028 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:07:59.215987 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a044d61c83513a339f04812741eddadaa69c2b5cba5bb9cfc9da626db038b0d3"} err="failed to get container status \"a044d61c83513a339f04812741eddadaa69c2b5cba5bb9cfc9da626db038b0d3\": rpc error: code = NotFound desc = could not find container \"a044d61c83513a339f04812741eddadaa69c2b5cba5bb9cfc9da626db038b0d3\": container with ID starting with a044d61c83513a339f04812741eddadaa69c2b5cba5bb9cfc9da626db038b0d3 not found: ID does not exist" Jul 07 16:07:59.232449 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:07:59.232428 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-ait-6eebde10-64d664cd44-4t744"] Jul 07 16:07:59.240424 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:07:59.240406 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-ait-6eebde10-64d664cd44-4t744"] Jul 07 16:07:59.494230 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:07:59.494199 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:07:59.663151 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:07:59.662766 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8d65757f-db0f-4b2a-893e-279002e8452a" path="/var/lib/kubelet/pods/8d65757f-db0f-4b2a-893e-279002e8452a/volumes" Jul 07 16:08:03.588228 ip-10-0-132-65 kubenswrapper[2565]: W0707 16:08:03.588199 2565 container.go:598] Failed to update stats for container "/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6dc3c3ea_a70d_4ec9_bf5a_83613d8b45a8.slice/crio-a9f36d995d22c9e905b3f01f1b81722e4adb047855e5f7975f287fe154197b77": error while statting cgroup v2: [openat2 /sys/fs/cgroup/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6dc3c3ea_a70d_4ec9_bf5a_83613d8b45a8.slice/crio-a9f36d995d22c9e905b3f01f1b81722e4adb047855e5f7975f287fe154197b77/pids.current: no such device], continuing to push stats Jul 07 16:08:03.740261 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:03.740236 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-6eebde10-openshift-default-54868ffd59-5b2kt" Jul 07 16:08:03.898306 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:03.898264 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6dc3c3ea-a70d-4ec9-bf5a-83613d8b45a8-istio-token\") pod \"6dc3c3ea-a70d-4ec9-bf5a-83613d8b45a8\" (UID: \"6dc3c3ea-a70d-4ec9-bf5a-83613d8b45a8\") " Jul 07 16:08:03.898519 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:03.898334 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6dc3c3ea-a70d-4ec9-bf5a-83613d8b45a8-istio-podinfo\") pod \"6dc3c3ea-a70d-4ec9-bf5a-83613d8b45a8\" (UID: \"6dc3c3ea-a70d-4ec9-bf5a-83613d8b45a8\") " Jul 07 16:08:03.898519 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:03.898355 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6dc3c3ea-a70d-4ec9-bf5a-83613d8b45a8-istiod-ca-cert\") pod \"6dc3c3ea-a70d-4ec9-bf5a-83613d8b45a8\" (UID: \"6dc3c3ea-a70d-4ec9-bf5a-83613d8b45a8\") " Jul 07 16:08:03.898519 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:03.898405 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6dc3c3ea-a70d-4ec9-bf5a-83613d8b45a8-workload-certs\") pod \"6dc3c3ea-a70d-4ec9-bf5a-83613d8b45a8\" (UID: \"6dc3c3ea-a70d-4ec9-bf5a-83613d8b45a8\") " Jul 07 16:08:03.898519 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:03.898431 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-57whh\" (UniqueName: \"kubernetes.io/projected/6dc3c3ea-a70d-4ec9-bf5a-83613d8b45a8-kube-api-access-57whh\") pod \"6dc3c3ea-a70d-4ec9-bf5a-83613d8b45a8\" (UID: \"6dc3c3ea-a70d-4ec9-bf5a-83613d8b45a8\") " Jul 07 16:08:03.898519 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:03.898476 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dc3c3ea-a70d-4ec9-bf5a-83613d8b45a8-credential-socket\") pod \"6dc3c3ea-a70d-4ec9-bf5a-83613d8b45a8\" (UID: \"6dc3c3ea-a70d-4ec9-bf5a-83613d8b45a8\") " Jul 07 16:08:03.898779 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:03.898531 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dc3c3ea-a70d-4ec9-bf5a-83613d8b45a8-workload-socket\") pod \"6dc3c3ea-a70d-4ec9-bf5a-83613d8b45a8\" (UID: \"6dc3c3ea-a70d-4ec9-bf5a-83613d8b45a8\") " Jul 07 16:08:03.898779 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:03.898573 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6dc3c3ea-a70d-4ec9-bf5a-83613d8b45a8-istio-data\") pod \"6dc3c3ea-a70d-4ec9-bf5a-83613d8b45a8\" (UID: \"6dc3c3ea-a70d-4ec9-bf5a-83613d8b45a8\") " Jul 07 16:08:03.898779 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:03.898604 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6dc3c3ea-a70d-4ec9-bf5a-83613d8b45a8-istio-envoy\") pod \"6dc3c3ea-a70d-4ec9-bf5a-83613d8b45a8\" (UID: \"6dc3c3ea-a70d-4ec9-bf5a-83613d8b45a8\") " Jul 07 16:08:03.898779 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:03.898771 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6dc3c3ea-a70d-4ec9-bf5a-83613d8b45a8-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "6dc3c3ea-a70d-4ec9-bf5a-83613d8b45a8" (UID: "6dc3c3ea-a70d-4ec9-bf5a-83613d8b45a8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 16:08:03.899006 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:03.898905 2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6dc3c3ea-a70d-4ec9-bf5a-83613d8b45a8-workload-certs\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 07 16:08:03.899006 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:03.898960 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6dc3c3ea-a70d-4ec9-bf5a-83613d8b45a8-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "6dc3c3ea-a70d-4ec9-bf5a-83613d8b45a8" (UID: "6dc3c3ea-a70d-4ec9-bf5a-83613d8b45a8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 16:08:03.899006 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:03.898954 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6dc3c3ea-a70d-4ec9-bf5a-83613d8b45a8-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "6dc3c3ea-a70d-4ec9-bf5a-83613d8b45a8" (UID: "6dc3c3ea-a70d-4ec9-bf5a-83613d8b45a8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 16:08:03.899226 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:03.899185 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6dc3c3ea-a70d-4ec9-bf5a-83613d8b45a8-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "6dc3c3ea-a70d-4ec9-bf5a-83613d8b45a8" (UID: "6dc3c3ea-a70d-4ec9-bf5a-83613d8b45a8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 16:08:03.899510 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:03.899470 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6dc3c3ea-a70d-4ec9-bf5a-83613d8b45a8-istio-data" (OuterVolumeSpecName: "istio-data") pod "6dc3c3ea-a70d-4ec9-bf5a-83613d8b45a8" (UID: "6dc3c3ea-a70d-4ec9-bf5a-83613d8b45a8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 16:08:03.900667 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:03.900647 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6dc3c3ea-a70d-4ec9-bf5a-83613d8b45a8-istio-token" (OuterVolumeSpecName: "istio-token") pod "6dc3c3ea-a70d-4ec9-bf5a-83613d8b45a8" (UID: "6dc3c3ea-a70d-4ec9-bf5a-83613d8b45a8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 16:08:03.900756 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:03.900664 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6dc3c3ea-a70d-4ec9-bf5a-83613d8b45a8-kube-api-access-57whh" (OuterVolumeSpecName: "kube-api-access-57whh") pod "6dc3c3ea-a70d-4ec9-bf5a-83613d8b45a8" (UID: "6dc3c3ea-a70d-4ec9-bf5a-83613d8b45a8"). InnerVolumeSpecName "kube-api-access-57whh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 16:08:03.900963 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:03.900943 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6dc3c3ea-a70d-4ec9-bf5a-83613d8b45a8-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "6dc3c3ea-a70d-4ec9-bf5a-83613d8b45a8" (UID: "6dc3c3ea-a70d-4ec9-bf5a-83613d8b45a8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 07 16:08:03.901019 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:03.900952 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6dc3c3ea-a70d-4ec9-bf5a-83613d8b45a8-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "6dc3c3ea-a70d-4ec9-bf5a-83613d8b45a8" (UID: "6dc3c3ea-a70d-4ec9-bf5a-83613d8b45a8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 16:08:03.999614 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:03.999580 2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6dc3c3ea-a70d-4ec9-bf5a-83613d8b45a8-istio-data\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 07 16:08:03.999614 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:03.999609 2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6dc3c3ea-a70d-4ec9-bf5a-83613d8b45a8-istio-envoy\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 07 16:08:03.999614 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:03.999618 2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6dc3c3ea-a70d-4ec9-bf5a-83613d8b45a8-istio-token\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 07 16:08:03.999614 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:03.999627 2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6dc3c3ea-a70d-4ec9-bf5a-83613d8b45a8-istio-podinfo\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 07 16:08:03.999614 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:03.999635 2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6dc3c3ea-a70d-4ec9-bf5a-83613d8b45a8-istiod-ca-cert\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 07 16:08:03.999923 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:03.999644 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-57whh\" (UniqueName: \"kubernetes.io/projected/6dc3c3ea-a70d-4ec9-bf5a-83613d8b45a8-kube-api-access-57whh\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 07 16:08:03.999923 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:03.999654 2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dc3c3ea-a70d-4ec9-bf5a-83613d8b45a8-credential-socket\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 07 16:08:03.999923 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:03.999663 2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dc3c3ea-a70d-4ec9-bf5a-83613d8b45a8-workload-socket\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 07 16:08:04.230904 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:04.230817 2565 generic.go:356] "Generic (PLEG): container finished" podID="6dc3c3ea-a70d-4ec9-bf5a-83613d8b45a8" containerID="8f6f81623d9cb0ca963e55744731f4e2ba52091b80966f4d4758228ee5fc1a80" exitCode=0 Jul 07 16:08:04.230904 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:04.230864 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-6eebde10-openshift-default-54868ffd59-5b2kt" Jul 07 16:08:04.231142 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:04.230904 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-6eebde10-openshift-default-54868ffd59-5b2kt" event={"ID":"6dc3c3ea-a70d-4ec9-bf5a-83613d8b45a8","Type":"ContainerDied","Data":"8f6f81623d9cb0ca963e55744731f4e2ba52091b80966f4d4758228ee5fc1a80"} Jul 07 16:08:04.231142 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:04.230970 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-6eebde10-openshift-default-54868ffd59-5b2kt" event={"ID":"6dc3c3ea-a70d-4ec9-bf5a-83613d8b45a8","Type":"ContainerDied","Data":"a9f36d995d22c9e905b3f01f1b81722e4adb047855e5f7975f287fe154197b77"} Jul 07 16:08:04.231142 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:04.230999 2565 scope.go:117] "RemoveContainer" containerID="8f6f81623d9cb0ca963e55744731f4e2ba52091b80966f4d4758228ee5fc1a80" Jul 07 16:08:04.240778 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:04.240759 2565 scope.go:117] "RemoveContainer" containerID="8f6f81623d9cb0ca963e55744731f4e2ba52091b80966f4d4758228ee5fc1a80" Jul 07 16:08:04.241025 ip-10-0-132-65 kubenswrapper[2565]: E0707 16:08:04.241004 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f6f81623d9cb0ca963e55744731f4e2ba52091b80966f4d4758228ee5fc1a80\": container with ID starting with 8f6f81623d9cb0ca963e55744731f4e2ba52091b80966f4d4758228ee5fc1a80 not found: ID does not exist" containerID="8f6f81623d9cb0ca963e55744731f4e2ba52091b80966f4d4758228ee5fc1a80" Jul 07 16:08:04.241076 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:04.241034 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f6f81623d9cb0ca963e55744731f4e2ba52091b80966f4d4758228ee5fc1a80"} err="failed to get container status \"8f6f81623d9cb0ca963e55744731f4e2ba52091b80966f4d4758228ee5fc1a80\": rpc error: code = NotFound desc = could not find container \"8f6f81623d9cb0ca963e55744731f4e2ba52091b80966f4d4758228ee5fc1a80\": container with ID starting with 8f6f81623d9cb0ca963e55744731f4e2ba52091b80966f4d4758228ee5fc1a80 not found: ID does not exist" Jul 07 16:08:04.274757 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:04.274727 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-6eebde10-openshift-default-54868ffd59-5b2kt"] Jul 07 16:08:04.303907 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:04.303877 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-6eebde10-openshift-default-54868ffd59-5b2kt"] Jul 07 16:08:05.449366 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:05.449334 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:08:05.549303 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:05.549264 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:08:05.663763 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:05.663727 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6dc3c3ea-a70d-4ec9-bf5a-83613d8b45a8" path="/var/lib/kubelet/pods/6dc3c3ea-a70d-4ec9-bf5a-83613d8b45a8/volumes" Jul 07 16:08:05.664037 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:05.664021 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-4bf95acf-openshift-default-fb9675fc9-j6z9t"] Jul 07 16:08:05.664400 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:05.664387 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d65757f-db0f-4b2a-893e-279002e8452a" containerName="maas-api" Jul 07 16:08:05.664465 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:05.664401 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d65757f-db0f-4b2a-893e-279002e8452a" containerName="maas-api" Jul 07 16:08:05.664465 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:05.664417 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6dc3c3ea-a70d-4ec9-bf5a-83613d8b45a8" containerName="istio-proxy" Jul 07 16:08:05.664465 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:05.664423 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="6dc3c3ea-a70d-4ec9-bf5a-83613d8b45a8" containerName="istio-proxy" Jul 07 16:08:05.664639 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:05.664521 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="6dc3c3ea-a70d-4ec9-bf5a-83613d8b45a8" containerName="istio-proxy" Jul 07 16:08:05.664639 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:05.664533 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="8d65757f-db0f-4b2a-893e-279002e8452a" containerName="maas-api" Jul 07 16:08:05.667395 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:05.667375 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-4bf95acf-openshift-default-fb9675fc9-j6z9t" Jul 07 16:08:05.670821 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:05.670802 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-4bf95acf-openshift-default-dockercfg-r9rtr\"" Jul 07 16:08:05.684038 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:05.684014 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-4bf95acf-openshift-default-fb9675fc9-j6z9t"] Jul 07 16:08:05.711178 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:05.711091 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:08:05.806896 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:05.806859 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:08:05.817082 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:05.817060 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd5ea175-1b85-49be-af99-49ee6db8e04b-istio-envoy\") pod \"e2e-ait-4bf95acf-openshift-default-fb9675fc9-j6z9t\" (UID: \"fd5ea175-1b85-49be-af99-49ee6db8e04b\") " pod="openshift-ingress/e2e-ait-4bf95acf-openshift-default-fb9675fc9-j6z9t" Jul 07 16:08:05.817205 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:05.817090 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bp59g\" (UniqueName: \"kubernetes.io/projected/fd5ea175-1b85-49be-af99-49ee6db8e04b-kube-api-access-bp59g\") pod \"e2e-ait-4bf95acf-openshift-default-fb9675fc9-j6z9t\" (UID: \"fd5ea175-1b85-49be-af99-49ee6db8e04b\") " pod="openshift-ingress/e2e-ait-4bf95acf-openshift-default-fb9675fc9-j6z9t" Jul 07 16:08:05.817205 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:05.817124 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd5ea175-1b85-49be-af99-49ee6db8e04b-istio-data\") pod \"e2e-ait-4bf95acf-openshift-default-fb9675fc9-j6z9t\" (UID: \"fd5ea175-1b85-49be-af99-49ee6db8e04b\") " pod="openshift-ingress/e2e-ait-4bf95acf-openshift-default-fb9675fc9-j6z9t" Jul 07 16:08:05.817205 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:05.817165 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd5ea175-1b85-49be-af99-49ee6db8e04b-istio-podinfo\") pod \"e2e-ait-4bf95acf-openshift-default-fb9675fc9-j6z9t\" (UID: \"fd5ea175-1b85-49be-af99-49ee6db8e04b\") " pod="openshift-ingress/e2e-ait-4bf95acf-openshift-default-fb9675fc9-j6z9t" Jul 07 16:08:05.817205 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:05.817190 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd5ea175-1b85-49be-af99-49ee6db8e04b-istiod-ca-cert\") pod \"e2e-ait-4bf95acf-openshift-default-fb9675fc9-j6z9t\" (UID: \"fd5ea175-1b85-49be-af99-49ee6db8e04b\") " pod="openshift-ingress/e2e-ait-4bf95acf-openshift-default-fb9675fc9-j6z9t" Jul 07 16:08:05.817465 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:05.817332 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd5ea175-1b85-49be-af99-49ee6db8e04b-workload-socket\") pod \"e2e-ait-4bf95acf-openshift-default-fb9675fc9-j6z9t\" (UID: \"fd5ea175-1b85-49be-af99-49ee6db8e04b\") " pod="openshift-ingress/e2e-ait-4bf95acf-openshift-default-fb9675fc9-j6z9t" Jul 07 16:08:05.817465 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:05.817371 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd5ea175-1b85-49be-af99-49ee6db8e04b-credential-socket\") pod \"e2e-ait-4bf95acf-openshift-default-fb9675fc9-j6z9t\" (UID: \"fd5ea175-1b85-49be-af99-49ee6db8e04b\") " pod="openshift-ingress/e2e-ait-4bf95acf-openshift-default-fb9675fc9-j6z9t" Jul 07 16:08:05.817465 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:05.817437 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd5ea175-1b85-49be-af99-49ee6db8e04b-workload-certs\") pod \"e2e-ait-4bf95acf-openshift-default-fb9675fc9-j6z9t\" (UID: \"fd5ea175-1b85-49be-af99-49ee6db8e04b\") " pod="openshift-ingress/e2e-ait-4bf95acf-openshift-default-fb9675fc9-j6z9t" Jul 07 16:08:05.817613 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:05.817466 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd5ea175-1b85-49be-af99-49ee6db8e04b-istio-token\") pod \"e2e-ait-4bf95acf-openshift-default-fb9675fc9-j6z9t\" (UID: \"fd5ea175-1b85-49be-af99-49ee6db8e04b\") " pod="openshift-ingress/e2e-ait-4bf95acf-openshift-default-fb9675fc9-j6z9t" Jul 07 16:08:05.918367 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:05.918327 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd5ea175-1b85-49be-af99-49ee6db8e04b-istio-envoy\") pod \"e2e-ait-4bf95acf-openshift-default-fb9675fc9-j6z9t\" (UID: \"fd5ea175-1b85-49be-af99-49ee6db8e04b\") " pod="openshift-ingress/e2e-ait-4bf95acf-openshift-default-fb9675fc9-j6z9t" Jul 07 16:08:05.918367 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:05.918364 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bp59g\" (UniqueName: \"kubernetes.io/projected/fd5ea175-1b85-49be-af99-49ee6db8e04b-kube-api-access-bp59g\") pod \"e2e-ait-4bf95acf-openshift-default-fb9675fc9-j6z9t\" (UID: \"fd5ea175-1b85-49be-af99-49ee6db8e04b\") " pod="openshift-ingress/e2e-ait-4bf95acf-openshift-default-fb9675fc9-j6z9t" Jul 07 16:08:05.918606 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:05.918385 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd5ea175-1b85-49be-af99-49ee6db8e04b-istio-data\") pod \"e2e-ait-4bf95acf-openshift-default-fb9675fc9-j6z9t\" (UID: \"fd5ea175-1b85-49be-af99-49ee6db8e04b\") " pod="openshift-ingress/e2e-ait-4bf95acf-openshift-default-fb9675fc9-j6z9t" Jul 07 16:08:05.918606 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:05.918515 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd5ea175-1b85-49be-af99-49ee6db8e04b-istio-podinfo\") pod \"e2e-ait-4bf95acf-openshift-default-fb9675fc9-j6z9t\" (UID: \"fd5ea175-1b85-49be-af99-49ee6db8e04b\") " pod="openshift-ingress/e2e-ait-4bf95acf-openshift-default-fb9675fc9-j6z9t" Jul 07 16:08:05.918606 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:05.918549 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd5ea175-1b85-49be-af99-49ee6db8e04b-istiod-ca-cert\") pod \"e2e-ait-4bf95acf-openshift-default-fb9675fc9-j6z9t\" (UID: \"fd5ea175-1b85-49be-af99-49ee6db8e04b\") " pod="openshift-ingress/e2e-ait-4bf95acf-openshift-default-fb9675fc9-j6z9t" Jul 07 16:08:05.918738 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:05.918640 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd5ea175-1b85-49be-af99-49ee6db8e04b-workload-socket\") pod \"e2e-ait-4bf95acf-openshift-default-fb9675fc9-j6z9t\" (UID: \"fd5ea175-1b85-49be-af99-49ee6db8e04b\") " pod="openshift-ingress/e2e-ait-4bf95acf-openshift-default-fb9675fc9-j6z9t" Jul 07 16:08:05.918738 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:05.918676 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd5ea175-1b85-49be-af99-49ee6db8e04b-credential-socket\") pod \"e2e-ait-4bf95acf-openshift-default-fb9675fc9-j6z9t\" (UID: \"fd5ea175-1b85-49be-af99-49ee6db8e04b\") " pod="openshift-ingress/e2e-ait-4bf95acf-openshift-default-fb9675fc9-j6z9t" Jul 07 16:08:05.918838 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:05.918781 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd5ea175-1b85-49be-af99-49ee6db8e04b-workload-certs\") pod \"e2e-ait-4bf95acf-openshift-default-fb9675fc9-j6z9t\" (UID: \"fd5ea175-1b85-49be-af99-49ee6db8e04b\") " pod="openshift-ingress/e2e-ait-4bf95acf-openshift-default-fb9675fc9-j6z9t" Jul 07 16:08:05.918838 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:05.918824 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd5ea175-1b85-49be-af99-49ee6db8e04b-istio-token\") pod \"e2e-ait-4bf95acf-openshift-default-fb9675fc9-j6z9t\" (UID: \"fd5ea175-1b85-49be-af99-49ee6db8e04b\") " pod="openshift-ingress/e2e-ait-4bf95acf-openshift-default-fb9675fc9-j6z9t" Jul 07 16:08:05.918952 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:05.918901 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd5ea175-1b85-49be-af99-49ee6db8e04b-istio-data\") pod \"e2e-ait-4bf95acf-openshift-default-fb9675fc9-j6z9t\" (UID: \"fd5ea175-1b85-49be-af99-49ee6db8e04b\") " pod="openshift-ingress/e2e-ait-4bf95acf-openshift-default-fb9675fc9-j6z9t" Jul 07 16:08:05.919009 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:05.918960 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd5ea175-1b85-49be-af99-49ee6db8e04b-workload-socket\") pod \"e2e-ait-4bf95acf-openshift-default-fb9675fc9-j6z9t\" (UID: \"fd5ea175-1b85-49be-af99-49ee6db8e04b\") " pod="openshift-ingress/e2e-ait-4bf95acf-openshift-default-fb9675fc9-j6z9t" Jul 07 16:08:05.919070 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:05.919012 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd5ea175-1b85-49be-af99-49ee6db8e04b-credential-socket\") pod \"e2e-ait-4bf95acf-openshift-default-fb9675fc9-j6z9t\" (UID: \"fd5ea175-1b85-49be-af99-49ee6db8e04b\") " pod="openshift-ingress/e2e-ait-4bf95acf-openshift-default-fb9675fc9-j6z9t" Jul 07 16:08:05.919150 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:05.919131 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd5ea175-1b85-49be-af99-49ee6db8e04b-workload-certs\") pod \"e2e-ait-4bf95acf-openshift-default-fb9675fc9-j6z9t\" (UID: \"fd5ea175-1b85-49be-af99-49ee6db8e04b\") " pod="openshift-ingress/e2e-ait-4bf95acf-openshift-default-fb9675fc9-j6z9t" Jul 07 16:08:05.919357 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:05.919337 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd5ea175-1b85-49be-af99-49ee6db8e04b-istiod-ca-cert\") pod \"e2e-ait-4bf95acf-openshift-default-fb9675fc9-j6z9t\" (UID: \"fd5ea175-1b85-49be-af99-49ee6db8e04b\") " pod="openshift-ingress/e2e-ait-4bf95acf-openshift-default-fb9675fc9-j6z9t" Jul 07 16:08:05.920623 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:05.920602 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd5ea175-1b85-49be-af99-49ee6db8e04b-istio-envoy\") pod \"e2e-ait-4bf95acf-openshift-default-fb9675fc9-j6z9t\" (UID: \"fd5ea175-1b85-49be-af99-49ee6db8e04b\") " pod="openshift-ingress/e2e-ait-4bf95acf-openshift-default-fb9675fc9-j6z9t" Jul 07 16:08:05.920888 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:05.920871 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd5ea175-1b85-49be-af99-49ee6db8e04b-istio-podinfo\") pod \"e2e-ait-4bf95acf-openshift-default-fb9675fc9-j6z9t\" (UID: \"fd5ea175-1b85-49be-af99-49ee6db8e04b\") " pod="openshift-ingress/e2e-ait-4bf95acf-openshift-default-fb9675fc9-j6z9t" Jul 07 16:08:05.939417 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:05.939394 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd5ea175-1b85-49be-af99-49ee6db8e04b-istio-token\") pod \"e2e-ait-4bf95acf-openshift-default-fb9675fc9-j6z9t\" (UID: \"fd5ea175-1b85-49be-af99-49ee6db8e04b\") " pod="openshift-ingress/e2e-ait-4bf95acf-openshift-default-fb9675fc9-j6z9t" Jul 07 16:08:05.939747 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:05.939725 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bp59g\" (UniqueName: \"kubernetes.io/projected/fd5ea175-1b85-49be-af99-49ee6db8e04b-kube-api-access-bp59g\") pod \"e2e-ait-4bf95acf-openshift-default-fb9675fc9-j6z9t\" (UID: \"fd5ea175-1b85-49be-af99-49ee6db8e04b\") " pod="openshift-ingress/e2e-ait-4bf95acf-openshift-default-fb9675fc9-j6z9t" Jul 07 16:08:05.977751 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:05.977665 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-4bf95acf-openshift-default-fb9675fc9-j6z9t" Jul 07 16:08:06.122167 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:06.122131 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-4bf95acf-openshift-default-fb9675fc9-j6z9t"] Jul 07 16:08:06.125479 ip-10-0-132-65 kubenswrapper[2565]: W0707 16:08:06.125448 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd5ea175_1b85_49be_af99_49ee6db8e04b.slice/crio-f3a6920259f814cab294aa4347b9a783763e12d3af5c6bae42685f8b2f4b12b9 WatchSource:0}: Error finding container f3a6920259f814cab294aa4347b9a783763e12d3af5c6bae42685f8b2f4b12b9: Status 404 returned error can't find the container with id f3a6920259f814cab294aa4347b9a783763e12d3af5c6bae42685f8b2f4b12b9 Jul 07 16:08:06.127594 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:06.127561 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 07 16:08:06.127664 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:06.127646 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 07 16:08:06.127707 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:06.127678 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 07 16:08:06.242541 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:06.242434 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-4bf95acf-openshift-default-fb9675fc9-j6z9t" event={"ID":"fd5ea175-1b85-49be-af99-49ee6db8e04b","Type":"ContainerStarted","Data":"1597ade95f7394375dba07104a336bc827a16ff7660974b2b3922be4788d186d"} Jul 07 16:08:06.242541 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:06.242503 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-4bf95acf-openshift-default-fb9675fc9-j6z9t" event={"ID":"fd5ea175-1b85-49be-af99-49ee6db8e04b","Type":"ContainerStarted","Data":"f3a6920259f814cab294aa4347b9a783763e12d3af5c6bae42685f8b2f4b12b9"} Jul 07 16:08:06.269292 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:06.269232 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-4bf95acf-openshift-default-fb9675fc9-j6z9t" podStartSLOduration=1.269209544 podStartE2EDuration="1.269209544s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:08:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:08:06.267353157 +0000 UTC m=+1625.223930624" watchObservedRunningTime="2026-07-07 16:08:06.269209544 +0000 UTC m=+1625.225787012" Jul 07 16:08:06.810501 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:06.810467 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:08:06.885577 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:06.885548 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-ait-4bf95acf-f477f7d4-5dr7l"] Jul 07 16:08:06.888124 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:06.888100 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-4bf95acf-f477f7d4-5dr7l" Jul 07 16:08:06.891214 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:06.891192 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-ait-4bf95acf-serving-cert\"" Jul 07 16:08:06.891683 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:06.891661 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-dockercfg-9bqq6\"" Jul 07 16:08:06.906394 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:06.906367 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-ait-4bf95acf-f477f7d4-5dr7l"] Jul 07 16:08:06.908815 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:06.908774 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:08:06.977922 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:06.977896 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-4bf95acf-openshift-default-fb9675fc9-j6z9t" Jul 07 16:08:07.012408 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:07.012373 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:08:07.027340 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:07.027312 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8aa6c46-ff85-4d44-b52f-a543e411bedf-openshift-service-ca\") pod \"maas-api-e2e-ait-4bf95acf-f477f7d4-5dr7l\" (UID: \"f8aa6c46-ff85-4d44-b52f-a543e411bedf\") " pod="opendatahub/maas-api-e2e-ait-4bf95acf-f477f7d4-5dr7l" Jul 07 16:08:07.027483 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:07.027406 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wzvr\" (UniqueName: \"kubernetes.io/projected/f8aa6c46-ff85-4d44-b52f-a543e411bedf-kube-api-access-8wzvr\") pod \"maas-api-e2e-ait-4bf95acf-f477f7d4-5dr7l\" (UID: \"f8aa6c46-ff85-4d44-b52f-a543e411bedf\") " pod="opendatahub/maas-api-e2e-ait-4bf95acf-f477f7d4-5dr7l" Jul 07 16:08:07.027483 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:07.027440 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f8aa6c46-ff85-4d44-b52f-a543e411bedf-maas-api-tls\") pod \"maas-api-e2e-ait-4bf95acf-f477f7d4-5dr7l\" (UID: \"f8aa6c46-ff85-4d44-b52f-a543e411bedf\") " pod="opendatahub/maas-api-e2e-ait-4bf95acf-f477f7d4-5dr7l" Jul 07 16:08:07.128841 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:07.128738 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f8aa6c46-ff85-4d44-b52f-a543e411bedf-maas-api-tls\") pod \"maas-api-e2e-ait-4bf95acf-f477f7d4-5dr7l\" (UID: \"f8aa6c46-ff85-4d44-b52f-a543e411bedf\") " pod="opendatahub/maas-api-e2e-ait-4bf95acf-f477f7d4-5dr7l" Jul 07 16:08:07.128841 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:07.128789 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8aa6c46-ff85-4d44-b52f-a543e411bedf-openshift-service-ca\") pod \"maas-api-e2e-ait-4bf95acf-f477f7d4-5dr7l\" (UID: \"f8aa6c46-ff85-4d44-b52f-a543e411bedf\") " pod="opendatahub/maas-api-e2e-ait-4bf95acf-f477f7d4-5dr7l" Jul 07 16:08:07.129094 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:07.128864 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wzvr\" (UniqueName: \"kubernetes.io/projected/f8aa6c46-ff85-4d44-b52f-a543e411bedf-kube-api-access-8wzvr\") pod \"maas-api-e2e-ait-4bf95acf-f477f7d4-5dr7l\" (UID: \"f8aa6c46-ff85-4d44-b52f-a543e411bedf\") " pod="opendatahub/maas-api-e2e-ait-4bf95acf-f477f7d4-5dr7l" Jul 07 16:08:07.129436 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:07.129414 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8aa6c46-ff85-4d44-b52f-a543e411bedf-openshift-service-ca\") pod \"maas-api-e2e-ait-4bf95acf-f477f7d4-5dr7l\" (UID: \"f8aa6c46-ff85-4d44-b52f-a543e411bedf\") " pod="opendatahub/maas-api-e2e-ait-4bf95acf-f477f7d4-5dr7l" Jul 07 16:08:07.131390 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:07.131367 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f8aa6c46-ff85-4d44-b52f-a543e411bedf-maas-api-tls\") pod \"maas-api-e2e-ait-4bf95acf-f477f7d4-5dr7l\" (UID: \"f8aa6c46-ff85-4d44-b52f-a543e411bedf\") " pod="opendatahub/maas-api-e2e-ait-4bf95acf-f477f7d4-5dr7l" Jul 07 16:08:07.142981 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:07.142956 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wzvr\" (UniqueName: \"kubernetes.io/projected/f8aa6c46-ff85-4d44-b52f-a543e411bedf-kube-api-access-8wzvr\") pod \"maas-api-e2e-ait-4bf95acf-f477f7d4-5dr7l\" (UID: \"f8aa6c46-ff85-4d44-b52f-a543e411bedf\") " pod="opendatahub/maas-api-e2e-ait-4bf95acf-f477f7d4-5dr7l" Jul 07 16:08:07.199158 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:07.199127 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-4bf95acf-f477f7d4-5dr7l" Jul 07 16:08:07.199368 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:07.199130 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:08:07.352419 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:07.352385 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-ait-4bf95acf-f477f7d4-5dr7l"] Jul 07 16:08:07.363942 ip-10-0-132-65 kubenswrapper[2565]: W0707 16:08:07.363914 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8aa6c46_ff85_4d44_b52f_a543e411bedf.slice/crio-9e82bae3e392ae199b3601a34890702bfd50d4b96ee0b2988b3277e41775fa9a WatchSource:0}: Error finding container 9e82bae3e392ae199b3601a34890702bfd50d4b96ee0b2988b3277e41775fa9a: Status 404 returned error can't find the container with id 9e82bae3e392ae199b3601a34890702bfd50d4b96ee0b2988b3277e41775fa9a Jul 07 16:08:07.816458 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:07.816421 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:08:07.921683 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:07.921646 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:08:07.992593 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:07.992509 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-4bf95acf-openshift-default-fb9675fc9-j6z9t" Jul 07 16:08:08.255338 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:08.255241 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-4bf95acf-f477f7d4-5dr7l" event={"ID":"f8aa6c46-ff85-4d44-b52f-a543e411bedf","Type":"ContainerStarted","Data":"6dc0ad5908cb5edcb5b66d9e0dd80cd78b582c8133517e4a812b48edd5bbd441"} Jul 07 16:08:08.255338 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:08.255305 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-4bf95acf-f477f7d4-5dr7l" event={"ID":"f8aa6c46-ff85-4d44-b52f-a543e411bedf","Type":"ContainerStarted","Data":"9e82bae3e392ae199b3601a34890702bfd50d4b96ee0b2988b3277e41775fa9a"} Jul 07 16:08:08.255598 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:08.255504 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-ait-4bf95acf-f477f7d4-5dr7l" Jul 07 16:08:08.255598 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:08.255552 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-4bf95acf-openshift-default-fb9675fc9-j6z9t" Jul 07 16:08:08.256476 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:08.256462 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-4bf95acf-openshift-default-fb9675fc9-j6z9t" Jul 07 16:08:08.281328 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:08.281273 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-ait-4bf95acf-f477f7d4-5dr7l" podStartSLOduration=1.9715205390000001 podStartE2EDuration="2.281255989s" totalImagesPullingTime="309.73545ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:08:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:08:08.279757474 +0000 UTC m=+1627.236334942" watchObservedRunningTime="2026-07-07 16:08:08.281255989 +0000 UTC m=+1627.237833495" Jul 07 16:08:09.180447 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:09.180373 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:08:12.043242 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:12.043206 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:08:12.751974 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:12.751940 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-ait-4bf95acf-f477f7d4-5dr7l"] Jul 07 16:08:12.752208 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:12.752176 2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-ait-4bf95acf-f477f7d4-5dr7l" podUID="f8aa6c46-ff85-4d44-b52f-a543e411bedf" containerName="maas-api" containerID="cri-o://6dc0ad5908cb5edcb5b66d9e0dd80cd78b582c8133517e4a812b48edd5bbd441" gracePeriod=30 Jul 07 16:08:12.757266 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:12.757240 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-ait-4bf95acf-f477f7d4-5dr7l" Jul 07 16:08:12.780126 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:12.780086 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:08:13.008885 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:13.008818 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-4bf95acf-f477f7d4-5dr7l" Jul 07 16:08:13.185867 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:13.185831 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8aa6c46-ff85-4d44-b52f-a543e411bedf-openshift-service-ca\") pod \"f8aa6c46-ff85-4d44-b52f-a543e411bedf\" (UID: \"f8aa6c46-ff85-4d44-b52f-a543e411bedf\") " Jul 07 16:08:13.186268 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:13.185890 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8wzvr\" (UniqueName: \"kubernetes.io/projected/f8aa6c46-ff85-4d44-b52f-a543e411bedf-kube-api-access-8wzvr\") pod \"f8aa6c46-ff85-4d44-b52f-a543e411bedf\" (UID: \"f8aa6c46-ff85-4d44-b52f-a543e411bedf\") " Jul 07 16:08:13.186268 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:13.185973 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f8aa6c46-ff85-4d44-b52f-a543e411bedf-maas-api-tls\") pod \"f8aa6c46-ff85-4d44-b52f-a543e411bedf\" (UID: \"f8aa6c46-ff85-4d44-b52f-a543e411bedf\") " Jul 07 16:08:13.186268 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:13.186254 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8aa6c46-ff85-4d44-b52f-a543e411bedf-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "f8aa6c46-ff85-4d44-b52f-a543e411bedf" (UID: "f8aa6c46-ff85-4d44-b52f-a543e411bedf"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 16:08:13.187982 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:13.187953 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8aa6c46-ff85-4d44-b52f-a543e411bedf-kube-api-access-8wzvr" (OuterVolumeSpecName: "kube-api-access-8wzvr") pod "f8aa6c46-ff85-4d44-b52f-a543e411bedf" (UID: "f8aa6c46-ff85-4d44-b52f-a543e411bedf"). InnerVolumeSpecName "kube-api-access-8wzvr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 16:08:13.187982 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:13.187970 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8aa6c46-ff85-4d44-b52f-a543e411bedf-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "f8aa6c46-ff85-4d44-b52f-a543e411bedf" (UID: "f8aa6c46-ff85-4d44-b52f-a543e411bedf"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 16:08:13.280822 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:13.280725 2565 generic.go:356] "Generic (PLEG): container finished" podID="f8aa6c46-ff85-4d44-b52f-a543e411bedf" containerID="6dc0ad5908cb5edcb5b66d9e0dd80cd78b582c8133517e4a812b48edd5bbd441" exitCode=0 Jul 07 16:08:13.280822 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:13.280778 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-4bf95acf-f477f7d4-5dr7l" Jul 07 16:08:13.280822 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:13.280775 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-4bf95acf-f477f7d4-5dr7l" event={"ID":"f8aa6c46-ff85-4d44-b52f-a543e411bedf","Type":"ContainerDied","Data":"6dc0ad5908cb5edcb5b66d9e0dd80cd78b582c8133517e4a812b48edd5bbd441"} Jul 07 16:08:13.281086 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:13.280838 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-4bf95acf-f477f7d4-5dr7l" event={"ID":"f8aa6c46-ff85-4d44-b52f-a543e411bedf","Type":"ContainerDied","Data":"9e82bae3e392ae199b3601a34890702bfd50d4b96ee0b2988b3277e41775fa9a"} Jul 07 16:08:13.281086 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:13.280860 2565 scope.go:117] "RemoveContainer" containerID="6dc0ad5908cb5edcb5b66d9e0dd80cd78b582c8133517e4a812b48edd5bbd441" Jul 07 16:08:13.286916 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:13.286893 2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8aa6c46-ff85-4d44-b52f-a543e411bedf-openshift-service-ca\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 07 16:08:13.286916 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:13.286912 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8wzvr\" (UniqueName: \"kubernetes.io/projected/f8aa6c46-ff85-4d44-b52f-a543e411bedf-kube-api-access-8wzvr\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 07 16:08:13.286916 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:13.286922 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f8aa6c46-ff85-4d44-b52f-a543e411bedf-maas-api-tls\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 07 16:08:13.290245 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:13.290229 2565 scope.go:117] "RemoveContainer" containerID="6dc0ad5908cb5edcb5b66d9e0dd80cd78b582c8133517e4a812b48edd5bbd441" Jul 07 16:08:13.290512 ip-10-0-132-65 kubenswrapper[2565]: E0707 16:08:13.290470 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6dc0ad5908cb5edcb5b66d9e0dd80cd78b582c8133517e4a812b48edd5bbd441\": container with ID starting with 6dc0ad5908cb5edcb5b66d9e0dd80cd78b582c8133517e4a812b48edd5bbd441 not found: ID does not exist" containerID="6dc0ad5908cb5edcb5b66d9e0dd80cd78b582c8133517e4a812b48edd5bbd441" Jul 07 16:08:13.290562 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:13.290531 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6dc0ad5908cb5edcb5b66d9e0dd80cd78b582c8133517e4a812b48edd5bbd441"} err="failed to get container status \"6dc0ad5908cb5edcb5b66d9e0dd80cd78b582c8133517e4a812b48edd5bbd441\": rpc error: code = NotFound desc = could not find container \"6dc0ad5908cb5edcb5b66d9e0dd80cd78b582c8133517e4a812b48edd5bbd441\": container with ID starting with 6dc0ad5908cb5edcb5b66d9e0dd80cd78b582c8133517e4a812b48edd5bbd441 not found: ID does not exist" Jul 07 16:08:13.307802 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:13.307777 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-ait-4bf95acf-f477f7d4-5dr7l"] Jul 07 16:08:13.318260 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:13.318237 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-ait-4bf95acf-f477f7d4-5dr7l"] Jul 07 16:08:13.660395 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:13.660355 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f8aa6c46-ff85-4d44-b52f-a543e411bedf" path="/var/lib/kubelet/pods/f8aa6c46-ff85-4d44-b52f-a543e411bedf/volumes" Jul 07 16:08:18.246300 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:18.246271 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-4bf95acf-openshift-default-fb9675fc9-j6z9t"] Jul 07 16:08:18.246895 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:18.246462 2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-4bf95acf-openshift-default-fb9675fc9-j6z9t" podUID="fd5ea175-1b85-49be-af99-49ee6db8e04b" containerName="istio-proxy" containerID="cri-o://1597ade95f7394375dba07104a336bc827a16ff7660974b2b3922be4788d186d" gracePeriod=30 Jul 07 16:08:18.301104 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:18.301070 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:08:19.466901 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:19.466858 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:08:20.130148 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:20.130100 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:08:21.704376 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:21.704350 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-4bf95acf-openshift-default-fb9675fc9-j6z9t" Jul 07 16:08:21.758155 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:21.758065 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bp59g\" (UniqueName: \"kubernetes.io/projected/fd5ea175-1b85-49be-af99-49ee6db8e04b-kube-api-access-bp59g\") pod \"fd5ea175-1b85-49be-af99-49ee6db8e04b\" (UID: \"fd5ea175-1b85-49be-af99-49ee6db8e04b\") " Jul 07 16:08:21.758155 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:21.758129 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd5ea175-1b85-49be-af99-49ee6db8e04b-istio-envoy\") pod \"fd5ea175-1b85-49be-af99-49ee6db8e04b\" (UID: \"fd5ea175-1b85-49be-af99-49ee6db8e04b\") " Jul 07 16:08:21.758364 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:21.758161 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd5ea175-1b85-49be-af99-49ee6db8e04b-istio-podinfo\") pod \"fd5ea175-1b85-49be-af99-49ee6db8e04b\" (UID: \"fd5ea175-1b85-49be-af99-49ee6db8e04b\") " Jul 07 16:08:21.758364 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:21.758214 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd5ea175-1b85-49be-af99-49ee6db8e04b-workload-socket\") pod \"fd5ea175-1b85-49be-af99-49ee6db8e04b\" (UID: \"fd5ea175-1b85-49be-af99-49ee6db8e04b\") " Jul 07 16:08:21.758364 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:21.758235 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd5ea175-1b85-49be-af99-49ee6db8e04b-istio-token\") pod \"fd5ea175-1b85-49be-af99-49ee6db8e04b\" (UID: \"fd5ea175-1b85-49be-af99-49ee6db8e04b\") " Jul 07 16:08:21.758364 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:21.758272 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd5ea175-1b85-49be-af99-49ee6db8e04b-istio-data\") pod \"fd5ea175-1b85-49be-af99-49ee6db8e04b\" (UID: \"fd5ea175-1b85-49be-af99-49ee6db8e04b\") " Jul 07 16:08:21.758364 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:21.758314 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd5ea175-1b85-49be-af99-49ee6db8e04b-istiod-ca-cert\") pod \"fd5ea175-1b85-49be-af99-49ee6db8e04b\" (UID: \"fd5ea175-1b85-49be-af99-49ee6db8e04b\") " Jul 07 16:08:21.758364 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:21.758344 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd5ea175-1b85-49be-af99-49ee6db8e04b-credential-socket\") pod \"fd5ea175-1b85-49be-af99-49ee6db8e04b\" (UID: \"fd5ea175-1b85-49be-af99-49ee6db8e04b\") " Jul 07 16:08:21.758687 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:21.758369 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd5ea175-1b85-49be-af99-49ee6db8e04b-workload-certs\") pod \"fd5ea175-1b85-49be-af99-49ee6db8e04b\" (UID: \"fd5ea175-1b85-49be-af99-49ee6db8e04b\") " Jul 07 16:08:21.758687 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:21.758627 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd5ea175-1b85-49be-af99-49ee6db8e04b-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "fd5ea175-1b85-49be-af99-49ee6db8e04b" (UID: "fd5ea175-1b85-49be-af99-49ee6db8e04b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 16:08:21.758792 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:21.758715 2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd5ea175-1b85-49be-af99-49ee6db8e04b-workload-socket\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 07 16:08:21.759182 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:21.758900 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd5ea175-1b85-49be-af99-49ee6db8e04b-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "fd5ea175-1b85-49be-af99-49ee6db8e04b" (UID: "fd5ea175-1b85-49be-af99-49ee6db8e04b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 16:08:21.759182 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:21.759128 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd5ea175-1b85-49be-af99-49ee6db8e04b-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "fd5ea175-1b85-49be-af99-49ee6db8e04b" (UID: "fd5ea175-1b85-49be-af99-49ee6db8e04b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 16:08:21.759595 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:21.759562 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd5ea175-1b85-49be-af99-49ee6db8e04b-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "fd5ea175-1b85-49be-af99-49ee6db8e04b" (UID: "fd5ea175-1b85-49be-af99-49ee6db8e04b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 16:08:21.759792 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:21.759764 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd5ea175-1b85-49be-af99-49ee6db8e04b-istio-data" (OuterVolumeSpecName: "istio-data") pod "fd5ea175-1b85-49be-af99-49ee6db8e04b" (UID: "fd5ea175-1b85-49be-af99-49ee6db8e04b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 16:08:21.760667 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:21.760635 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fd5ea175-1b85-49be-af99-49ee6db8e04b-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "fd5ea175-1b85-49be-af99-49ee6db8e04b" (UID: "fd5ea175-1b85-49be-af99-49ee6db8e04b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 07 16:08:21.760775 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:21.760684 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd5ea175-1b85-49be-af99-49ee6db8e04b-istio-token" (OuterVolumeSpecName: "istio-token") pod "fd5ea175-1b85-49be-af99-49ee6db8e04b" (UID: "fd5ea175-1b85-49be-af99-49ee6db8e04b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 16:08:21.760832 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:21.760769 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd5ea175-1b85-49be-af99-49ee6db8e04b-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "fd5ea175-1b85-49be-af99-49ee6db8e04b" (UID: "fd5ea175-1b85-49be-af99-49ee6db8e04b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 16:08:21.760832 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:21.760774 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd5ea175-1b85-49be-af99-49ee6db8e04b-kube-api-access-bp59g" (OuterVolumeSpecName: "kube-api-access-bp59g") pod "fd5ea175-1b85-49be-af99-49ee6db8e04b" (UID: "fd5ea175-1b85-49be-af99-49ee6db8e04b"). InnerVolumeSpecName "kube-api-access-bp59g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 16:08:21.859271 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:21.859235 2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd5ea175-1b85-49be-af99-49ee6db8e04b-istio-token\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 07 16:08:21.859271 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:21.859265 2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd5ea175-1b85-49be-af99-49ee6db8e04b-istio-data\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 07 16:08:21.859271 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:21.859275 2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd5ea175-1b85-49be-af99-49ee6db8e04b-istiod-ca-cert\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 07 16:08:21.859271 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:21.859283 2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd5ea175-1b85-49be-af99-49ee6db8e04b-credential-socket\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 07 16:08:21.859591 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:21.859294 2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd5ea175-1b85-49be-af99-49ee6db8e04b-workload-certs\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 07 16:08:21.859591 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:21.859302 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bp59g\" (UniqueName: \"kubernetes.io/projected/fd5ea175-1b85-49be-af99-49ee6db8e04b-kube-api-access-bp59g\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 07 16:08:21.859591 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:21.859311 2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd5ea175-1b85-49be-af99-49ee6db8e04b-istio-envoy\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 07 16:08:21.859591 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:21.859319 2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd5ea175-1b85-49be-af99-49ee6db8e04b-istio-podinfo\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 07 16:08:22.325607 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:22.325561 2565 generic.go:356] "Generic (PLEG): container finished" podID="fd5ea175-1b85-49be-af99-49ee6db8e04b" containerID="1597ade95f7394375dba07104a336bc827a16ff7660974b2b3922be4788d186d" exitCode=0 Jul 07 16:08:22.325607 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:22.325605 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-4bf95acf-openshift-default-fb9675fc9-j6z9t" event={"ID":"fd5ea175-1b85-49be-af99-49ee6db8e04b","Type":"ContainerDied","Data":"1597ade95f7394375dba07104a336bc827a16ff7660974b2b3922be4788d186d"} Jul 07 16:08:22.325894 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:22.325611 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-4bf95acf-openshift-default-fb9675fc9-j6z9t" Jul 07 16:08:22.325894 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:22.325656 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-4bf95acf-openshift-default-fb9675fc9-j6z9t" event={"ID":"fd5ea175-1b85-49be-af99-49ee6db8e04b","Type":"ContainerDied","Data":"f3a6920259f814cab294aa4347b9a783763e12d3af5c6bae42685f8b2f4b12b9"} Jul 07 16:08:22.325894 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:22.325684 2565 scope.go:117] "RemoveContainer" containerID="1597ade95f7394375dba07104a336bc827a16ff7660974b2b3922be4788d186d" Jul 07 16:08:22.335758 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:22.335740 2565 scope.go:117] "RemoveContainer" containerID="1597ade95f7394375dba07104a336bc827a16ff7660974b2b3922be4788d186d" Jul 07 16:08:22.336031 ip-10-0-132-65 kubenswrapper[2565]: E0707 16:08:22.336012 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1597ade95f7394375dba07104a336bc827a16ff7660974b2b3922be4788d186d\": container with ID starting with 1597ade95f7394375dba07104a336bc827a16ff7660974b2b3922be4788d186d not found: ID does not exist" containerID="1597ade95f7394375dba07104a336bc827a16ff7660974b2b3922be4788d186d" Jul 07 16:08:22.336102 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:22.336039 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1597ade95f7394375dba07104a336bc827a16ff7660974b2b3922be4788d186d"} err="failed to get container status \"1597ade95f7394375dba07104a336bc827a16ff7660974b2b3922be4788d186d\": rpc error: code = NotFound desc = could not find container \"1597ade95f7394375dba07104a336bc827a16ff7660974b2b3922be4788d186d\": container with ID starting with 1597ade95f7394375dba07104a336bc827a16ff7660974b2b3922be4788d186d not found: ID does not exist" Jul 07 16:08:22.354884 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:22.354849 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-4bf95acf-openshift-default-fb9675fc9-j6z9t"] Jul 07 16:08:22.359539 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:22.359515 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-4bf95acf-openshift-default-fb9675fc9-j6z9t"] Jul 07 16:08:23.661102 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:23.661069 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fd5ea175-1b85-49be-af99-49ee6db8e04b" path="/var/lib/kubelet/pods/fd5ea175-1b85-49be-af99-49ee6db8e04b/volumes" Jul 07 16:08:25.613152 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:25.613108 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:08:25.707393 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:25.707353 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:08:25.856826 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:25.856794 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-af112f23-openshift-default-ccb75999-zcd8d"] Jul 07 16:08:25.857177 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:25.857164 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd5ea175-1b85-49be-af99-49ee6db8e04b" containerName="istio-proxy" Jul 07 16:08:25.857177 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:25.857178 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd5ea175-1b85-49be-af99-49ee6db8e04b" containerName="istio-proxy" Jul 07 16:08:25.857300 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:25.857193 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8aa6c46-ff85-4d44-b52f-a543e411bedf" containerName="maas-api" Jul 07 16:08:25.857300 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:25.857202 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8aa6c46-ff85-4d44-b52f-a543e411bedf" containerName="maas-api" Jul 07 16:08:25.857420 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:25.857331 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="f8aa6c46-ff85-4d44-b52f-a543e411bedf" containerName="maas-api" Jul 07 16:08:25.857420 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:25.857343 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="fd5ea175-1b85-49be-af99-49ee6db8e04b" containerName="istio-proxy" Jul 07 16:08:25.860454 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:25.860432 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-af112f23-openshift-default-ccb75999-zcd8d" Jul 07 16:08:25.863437 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:25.863379 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-af112f23-openshift-default-dockercfg-4cz9q\"" Jul 07 16:08:25.880796 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:25.880766 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-af112f23-openshift-default-ccb75999-zcd8d"] Jul 07 16:08:25.896975 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:25.896946 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75be73ad-40d7-4a58-b36d-8ababf74ccc6-istiod-ca-cert\") pod \"e2e-derive-af112f23-openshift-default-ccb75999-zcd8d\" (UID: \"75be73ad-40d7-4a58-b36d-8ababf74ccc6\") " pod="openshift-ingress/e2e-derive-af112f23-openshift-default-ccb75999-zcd8d" Jul 07 16:08:25.897168 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:25.897050 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8m2wb\" (UniqueName: \"kubernetes.io/projected/75be73ad-40d7-4a58-b36d-8ababf74ccc6-kube-api-access-8m2wb\") pod \"e2e-derive-af112f23-openshift-default-ccb75999-zcd8d\" (UID: \"75be73ad-40d7-4a58-b36d-8ababf74ccc6\") " pod="openshift-ingress/e2e-derive-af112f23-openshift-default-ccb75999-zcd8d" Jul 07 16:08:25.897168 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:25.897101 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75be73ad-40d7-4a58-b36d-8ababf74ccc6-credential-socket\") pod \"e2e-derive-af112f23-openshift-default-ccb75999-zcd8d\" (UID: \"75be73ad-40d7-4a58-b36d-8ababf74ccc6\") " pod="openshift-ingress/e2e-derive-af112f23-openshift-default-ccb75999-zcd8d" Jul 07 16:08:25.897168 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:25.897134 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75be73ad-40d7-4a58-b36d-8ababf74ccc6-istio-envoy\") pod \"e2e-derive-af112f23-openshift-default-ccb75999-zcd8d\" (UID: \"75be73ad-40d7-4a58-b36d-8ababf74ccc6\") " pod="openshift-ingress/e2e-derive-af112f23-openshift-default-ccb75999-zcd8d" Jul 07 16:08:25.897371 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:25.897215 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75be73ad-40d7-4a58-b36d-8ababf74ccc6-workload-certs\") pod \"e2e-derive-af112f23-openshift-default-ccb75999-zcd8d\" (UID: \"75be73ad-40d7-4a58-b36d-8ababf74ccc6\") " pod="openshift-ingress/e2e-derive-af112f23-openshift-default-ccb75999-zcd8d" Jul 07 16:08:25.897371 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:25.897265 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75be73ad-40d7-4a58-b36d-8ababf74ccc6-workload-socket\") pod \"e2e-derive-af112f23-openshift-default-ccb75999-zcd8d\" (UID: \"75be73ad-40d7-4a58-b36d-8ababf74ccc6\") " pod="openshift-ingress/e2e-derive-af112f23-openshift-default-ccb75999-zcd8d" Jul 07 16:08:25.897371 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:25.897292 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75be73ad-40d7-4a58-b36d-8ababf74ccc6-istio-podinfo\") pod \"e2e-derive-af112f23-openshift-default-ccb75999-zcd8d\" (UID: \"75be73ad-40d7-4a58-b36d-8ababf74ccc6\") " pod="openshift-ingress/e2e-derive-af112f23-openshift-default-ccb75999-zcd8d" Jul 07 16:08:25.897371 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:25.897326 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75be73ad-40d7-4a58-b36d-8ababf74ccc6-istio-data\") pod \"e2e-derive-af112f23-openshift-default-ccb75999-zcd8d\" (UID: \"75be73ad-40d7-4a58-b36d-8ababf74ccc6\") " pod="openshift-ingress/e2e-derive-af112f23-openshift-default-ccb75999-zcd8d" Jul 07 16:08:25.897371 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:25.897351 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75be73ad-40d7-4a58-b36d-8ababf74ccc6-istio-token\") pod \"e2e-derive-af112f23-openshift-default-ccb75999-zcd8d\" (UID: \"75be73ad-40d7-4a58-b36d-8ababf74ccc6\") " pod="openshift-ingress/e2e-derive-af112f23-openshift-default-ccb75999-zcd8d" Jul 07 16:08:25.898268 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:25.898245 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:08:25.995388 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:25.995345 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:08:25.998644 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:25.998625 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75be73ad-40d7-4a58-b36d-8ababf74ccc6-istio-envoy\") pod \"e2e-derive-af112f23-openshift-default-ccb75999-zcd8d\" (UID: \"75be73ad-40d7-4a58-b36d-8ababf74ccc6\") " pod="openshift-ingress/e2e-derive-af112f23-openshift-default-ccb75999-zcd8d" Jul 07 16:08:25.998745 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:25.998660 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75be73ad-40d7-4a58-b36d-8ababf74ccc6-workload-certs\") pod \"e2e-derive-af112f23-openshift-default-ccb75999-zcd8d\" (UID: \"75be73ad-40d7-4a58-b36d-8ababf74ccc6\") " pod="openshift-ingress/e2e-derive-af112f23-openshift-default-ccb75999-zcd8d" Jul 07 16:08:25.998745 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:25.998680 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75be73ad-40d7-4a58-b36d-8ababf74ccc6-workload-socket\") pod \"e2e-derive-af112f23-openshift-default-ccb75999-zcd8d\" (UID: \"75be73ad-40d7-4a58-b36d-8ababf74ccc6\") " pod="openshift-ingress/e2e-derive-af112f23-openshift-default-ccb75999-zcd8d" Jul 07 16:08:25.998745 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:25.998695 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75be73ad-40d7-4a58-b36d-8ababf74ccc6-istio-podinfo\") pod \"e2e-derive-af112f23-openshift-default-ccb75999-zcd8d\" (UID: \"75be73ad-40d7-4a58-b36d-8ababf74ccc6\") " pod="openshift-ingress/e2e-derive-af112f23-openshift-default-ccb75999-zcd8d" Jul 07 16:08:25.998897 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:25.998748 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75be73ad-40d7-4a58-b36d-8ababf74ccc6-istio-data\") pod \"e2e-derive-af112f23-openshift-default-ccb75999-zcd8d\" (UID: \"75be73ad-40d7-4a58-b36d-8ababf74ccc6\") " pod="openshift-ingress/e2e-derive-af112f23-openshift-default-ccb75999-zcd8d" Jul 07 16:08:25.998897 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:25.998783 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75be73ad-40d7-4a58-b36d-8ababf74ccc6-istio-token\") pod \"e2e-derive-af112f23-openshift-default-ccb75999-zcd8d\" (UID: \"75be73ad-40d7-4a58-b36d-8ababf74ccc6\") " pod="openshift-ingress/e2e-derive-af112f23-openshift-default-ccb75999-zcd8d" Jul 07 16:08:25.998897 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:25.998827 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75be73ad-40d7-4a58-b36d-8ababf74ccc6-istiod-ca-cert\") pod \"e2e-derive-af112f23-openshift-default-ccb75999-zcd8d\" (UID: \"75be73ad-40d7-4a58-b36d-8ababf74ccc6\") " pod="openshift-ingress/e2e-derive-af112f23-openshift-default-ccb75999-zcd8d" Jul 07 16:08:25.999043 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:25.998940 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8m2wb\" (UniqueName: \"kubernetes.io/projected/75be73ad-40d7-4a58-b36d-8ababf74ccc6-kube-api-access-8m2wb\") pod \"e2e-derive-af112f23-openshift-default-ccb75999-zcd8d\" (UID: \"75be73ad-40d7-4a58-b36d-8ababf74ccc6\") " pod="openshift-ingress/e2e-derive-af112f23-openshift-default-ccb75999-zcd8d" Jul 07 16:08:25.999043 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:25.998997 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75be73ad-40d7-4a58-b36d-8ababf74ccc6-credential-socket\") pod \"e2e-derive-af112f23-openshift-default-ccb75999-zcd8d\" (UID: \"75be73ad-40d7-4a58-b36d-8ababf74ccc6\") " pod="openshift-ingress/e2e-derive-af112f23-openshift-default-ccb75999-zcd8d" Jul 07 16:08:25.999148 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:25.999075 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75be73ad-40d7-4a58-b36d-8ababf74ccc6-workload-certs\") pod \"e2e-derive-af112f23-openshift-default-ccb75999-zcd8d\" (UID: \"75be73ad-40d7-4a58-b36d-8ababf74ccc6\") " pod="openshift-ingress/e2e-derive-af112f23-openshift-default-ccb75999-zcd8d" Jul 07 16:08:25.999148 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:25.999094 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75be73ad-40d7-4a58-b36d-8ababf74ccc6-workload-socket\") pod \"e2e-derive-af112f23-openshift-default-ccb75999-zcd8d\" (UID: \"75be73ad-40d7-4a58-b36d-8ababf74ccc6\") " pod="openshift-ingress/e2e-derive-af112f23-openshift-default-ccb75999-zcd8d" Jul 07 16:08:25.999261 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:25.999171 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75be73ad-40d7-4a58-b36d-8ababf74ccc6-istio-data\") pod \"e2e-derive-af112f23-openshift-default-ccb75999-zcd8d\" (UID: \"75be73ad-40d7-4a58-b36d-8ababf74ccc6\") " pod="openshift-ingress/e2e-derive-af112f23-openshift-default-ccb75999-zcd8d" Jul 07 16:08:25.999558 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:25.999534 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75be73ad-40d7-4a58-b36d-8ababf74ccc6-credential-socket\") pod \"e2e-derive-af112f23-openshift-default-ccb75999-zcd8d\" (UID: \"75be73ad-40d7-4a58-b36d-8ababf74ccc6\") " pod="openshift-ingress/e2e-derive-af112f23-openshift-default-ccb75999-zcd8d" Jul 07 16:08:25.999683 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:25.999663 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75be73ad-40d7-4a58-b36d-8ababf74ccc6-istiod-ca-cert\") pod \"e2e-derive-af112f23-openshift-default-ccb75999-zcd8d\" (UID: \"75be73ad-40d7-4a58-b36d-8ababf74ccc6\") " pod="openshift-ingress/e2e-derive-af112f23-openshift-default-ccb75999-zcd8d" Jul 07 16:08:26.001086 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:26.001065 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75be73ad-40d7-4a58-b36d-8ababf74ccc6-istio-envoy\") pod \"e2e-derive-af112f23-openshift-default-ccb75999-zcd8d\" (UID: \"75be73ad-40d7-4a58-b36d-8ababf74ccc6\") " pod="openshift-ingress/e2e-derive-af112f23-openshift-default-ccb75999-zcd8d" Jul 07 16:08:26.001242 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:26.001219 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75be73ad-40d7-4a58-b36d-8ababf74ccc6-istio-podinfo\") pod \"e2e-derive-af112f23-openshift-default-ccb75999-zcd8d\" (UID: \"75be73ad-40d7-4a58-b36d-8ababf74ccc6\") " pod="openshift-ingress/e2e-derive-af112f23-openshift-default-ccb75999-zcd8d" Jul 07 16:08:26.012241 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:26.012216 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75be73ad-40d7-4a58-b36d-8ababf74ccc6-istio-token\") pod \"e2e-derive-af112f23-openshift-default-ccb75999-zcd8d\" (UID: \"75be73ad-40d7-4a58-b36d-8ababf74ccc6\") " pod="openshift-ingress/e2e-derive-af112f23-openshift-default-ccb75999-zcd8d" Jul 07 16:08:26.012385 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:26.012360 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8m2wb\" (UniqueName: \"kubernetes.io/projected/75be73ad-40d7-4a58-b36d-8ababf74ccc6-kube-api-access-8m2wb\") pod \"e2e-derive-af112f23-openshift-default-ccb75999-zcd8d\" (UID: \"75be73ad-40d7-4a58-b36d-8ababf74ccc6\") " pod="openshift-ingress/e2e-derive-af112f23-openshift-default-ccb75999-zcd8d" Jul 07 16:08:26.171933 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:26.171848 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-af112f23-openshift-default-ccb75999-zcd8d" Jul 07 16:08:26.325997 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:26.325971 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-af112f23-openshift-default-ccb75999-zcd8d"] Jul 07 16:08:26.328369 ip-10-0-132-65 kubenswrapper[2565]: W0707 16:08:26.328343 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75be73ad_40d7_4a58_b36d_8ababf74ccc6.slice/crio-c4c99296bec43aa7d2e42dbde8527f04a2f4bf16dbbf0fed9bcc0d22e2d9c681 WatchSource:0}: Error finding container c4c99296bec43aa7d2e42dbde8527f04a2f4bf16dbbf0fed9bcc0d22e2d9c681: Status 404 returned error can't find the container with id c4c99296bec43aa7d2e42dbde8527f04a2f4bf16dbbf0fed9bcc0d22e2d9c681 Jul 07 16:08:26.330378 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:26.330350 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 07 16:08:26.330462 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:26.330417 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 07 16:08:26.330462 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:26.330449 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 07 16:08:26.346741 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:26.346711 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-af112f23-openshift-default-ccb75999-zcd8d" event={"ID":"75be73ad-40d7-4a58-b36d-8ababf74ccc6","Type":"ContainerStarted","Data":"c4c99296bec43aa7d2e42dbde8527f04a2f4bf16dbbf0fed9bcc0d22e2d9c681"} Jul 07 16:08:27.071893 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:27.071853 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:08:27.141918 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:27.141883 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-derive-af112f23-6c7f76b67c-zkq54"] Jul 07 16:08:27.144350 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:27.144333 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-derive-af112f23-6c7f76b67c-zkq54" Jul 07 16:08:27.147639 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:27.147614 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-derive-af112f23-serving-cert\"" Jul 07 16:08:27.150378 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:27.150355 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-dockercfg-9bqq6\"" Jul 07 16:08:27.162092 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:27.162058 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-derive-af112f23-6c7f76b67c-zkq54"] Jul 07 16:08:27.182841 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:27.182811 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:08:27.209113 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:27.209080 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9sn5g\" (UniqueName: \"kubernetes.io/projected/691e9fb7-eaa8-46a2-964e-8abe0d190fb4-kube-api-access-9sn5g\") pod \"maas-api-e2e-derive-af112f23-6c7f76b67c-zkq54\" (UID: \"691e9fb7-eaa8-46a2-964e-8abe0d190fb4\") " pod="opendatahub/maas-api-e2e-derive-af112f23-6c7f76b67c-zkq54" Jul 07 16:08:27.209299 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:27.209129 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/691e9fb7-eaa8-46a2-964e-8abe0d190fb4-maas-api-tls\") pod \"maas-api-e2e-derive-af112f23-6c7f76b67c-zkq54\" (UID: \"691e9fb7-eaa8-46a2-964e-8abe0d190fb4\") " pod="opendatahub/maas-api-e2e-derive-af112f23-6c7f76b67c-zkq54" Jul 07 16:08:27.209299 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:27.209219 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/691e9fb7-eaa8-46a2-964e-8abe0d190fb4-openshift-service-ca\") pod \"maas-api-e2e-derive-af112f23-6c7f76b67c-zkq54\" (UID: \"691e9fb7-eaa8-46a2-964e-8abe0d190fb4\") " pod="opendatahub/maas-api-e2e-derive-af112f23-6c7f76b67c-zkq54" Jul 07 16:08:27.299003 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:27.298959 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:08:27.309771 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:27.309741 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/691e9fb7-eaa8-46a2-964e-8abe0d190fb4-openshift-service-ca\") pod \"maas-api-e2e-derive-af112f23-6c7f76b67c-zkq54\" (UID: \"691e9fb7-eaa8-46a2-964e-8abe0d190fb4\") " pod="opendatahub/maas-api-e2e-derive-af112f23-6c7f76b67c-zkq54" Jul 07 16:08:27.309957 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:27.309853 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9sn5g\" (UniqueName: \"kubernetes.io/projected/691e9fb7-eaa8-46a2-964e-8abe0d190fb4-kube-api-access-9sn5g\") pod \"maas-api-e2e-derive-af112f23-6c7f76b67c-zkq54\" (UID: \"691e9fb7-eaa8-46a2-964e-8abe0d190fb4\") " pod="opendatahub/maas-api-e2e-derive-af112f23-6c7f76b67c-zkq54" Jul 07 16:08:27.309957 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:27.309878 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/691e9fb7-eaa8-46a2-964e-8abe0d190fb4-maas-api-tls\") pod \"maas-api-e2e-derive-af112f23-6c7f76b67c-zkq54\" (UID: \"691e9fb7-eaa8-46a2-964e-8abe0d190fb4\") " pod="opendatahub/maas-api-e2e-derive-af112f23-6c7f76b67c-zkq54" Jul 07 16:08:27.310464 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:27.310443 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/691e9fb7-eaa8-46a2-964e-8abe0d190fb4-openshift-service-ca\") pod \"maas-api-e2e-derive-af112f23-6c7f76b67c-zkq54\" (UID: \"691e9fb7-eaa8-46a2-964e-8abe0d190fb4\") " pod="opendatahub/maas-api-e2e-derive-af112f23-6c7f76b67c-zkq54" Jul 07 16:08:27.312275 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:27.312252 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/691e9fb7-eaa8-46a2-964e-8abe0d190fb4-maas-api-tls\") pod \"maas-api-e2e-derive-af112f23-6c7f76b67c-zkq54\" (UID: \"691e9fb7-eaa8-46a2-964e-8abe0d190fb4\") " pod="opendatahub/maas-api-e2e-derive-af112f23-6c7f76b67c-zkq54" Jul 07 16:08:27.330009 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:27.329938 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9sn5g\" (UniqueName: \"kubernetes.io/projected/691e9fb7-eaa8-46a2-964e-8abe0d190fb4-kube-api-access-9sn5g\") pod \"maas-api-e2e-derive-af112f23-6c7f76b67c-zkq54\" (UID: \"691e9fb7-eaa8-46a2-964e-8abe0d190fb4\") " pod="opendatahub/maas-api-e2e-derive-af112f23-6c7f76b67c-zkq54" Jul 07 16:08:27.353728 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:27.353700 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-af112f23-openshift-default-ccb75999-zcd8d" event={"ID":"75be73ad-40d7-4a58-b36d-8ababf74ccc6","Type":"ContainerStarted","Data":"214c97d0252d83475d862c5754d9bb61fd24af69de4ed0a687b8f1d03f1f460f"} Jul 07 16:08:27.392712 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:27.392675 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:08:27.456068 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:27.456034 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-derive-af112f23-6c7f76b67c-zkq54" Jul 07 16:08:27.610191 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:27.610111 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-af112f23-openshift-default-ccb75999-zcd8d" podStartSLOduration=2.610090661 podStartE2EDuration="2.610090661s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:08:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:08:27.381592221 +0000 UTC m=+1646.338169690" watchObservedRunningTime="2026-07-07 16:08:27.610090661 +0000 UTC m=+1646.566668128" Jul 07 16:08:27.612182 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:27.612158 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-derive-af112f23-6c7f76b67c-zkq54"] Jul 07 16:08:27.613350 ip-10-0-132-65 kubenswrapper[2565]: W0707 16:08:27.613325 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod691e9fb7_eaa8_46a2_964e_8abe0d190fb4.slice/crio-f4dbbd99f2264c4f1ab847e0ae064877bd3c9b822b4f321496dca0085ed39cf2 WatchSource:0}: Error finding container f4dbbd99f2264c4f1ab847e0ae064877bd3c9b822b4f321496dca0085ed39cf2: Status 404 returned error can't find the container with id f4dbbd99f2264c4f1ab847e0ae064877bd3c9b822b4f321496dca0085ed39cf2 Jul 07 16:08:27.696857 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:27.696795 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:08:27.808307 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:27.808270 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:08:28.172750 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:28.172713 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-af112f23-openshift-default-ccb75999-zcd8d" Jul 07 16:08:28.179094 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:28.179061 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-af112f23-openshift-default-ccb75999-zcd8d" Jul 07 16:08:28.361547 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:28.361482 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-derive-af112f23-6c7f76b67c-zkq54" event={"ID":"691e9fb7-eaa8-46a2-964e-8abe0d190fb4","Type":"ContainerStarted","Data":"77c4bb8565e2010b112c51b551fdbc1a4d154109d40aa36776faf7fd2baacb9e"} Jul 07 16:08:28.361729 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:28.361570 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-derive-af112f23-6c7f76b67c-zkq54" event={"ID":"691e9fb7-eaa8-46a2-964e-8abe0d190fb4","Type":"ContainerStarted","Data":"f4dbbd99f2264c4f1ab847e0ae064877bd3c9b822b4f321496dca0085ed39cf2"} Jul 07 16:08:28.361804 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:28.361782 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-derive-af112f23-6c7f76b67c-zkq54" Jul 07 16:08:28.361858 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:28.361833 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-af112f23-openshift-default-ccb75999-zcd8d" Jul 07 16:08:28.362831 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:28.362799 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-af112f23-openshift-default-ccb75999-zcd8d" Jul 07 16:08:28.385067 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:28.385018 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-derive-af112f23-6c7f76b67c-zkq54" podStartSLOduration=1.067123657 podStartE2EDuration="1.385001395s" totalImagesPullingTime="317.877738ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:08:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:08:28.382806407 +0000 UTC m=+1647.339383876" watchObservedRunningTime="2026-07-07 16:08:28.385001395 +0000 UTC m=+1647.341578862" Jul 07 16:08:28.989804 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:28.989762 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:08:31.626046 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:31.626005 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:08:31.775837 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:31.775765 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-af112f23-openshift-default-ccb75999-zcd8d"] Jul 07 16:08:31.776112 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:31.776063 2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-af112f23-openshift-default-ccb75999-zcd8d" podUID="75be73ad-40d7-4a58-b36d-8ababf74ccc6" containerName="istio-proxy" containerID="cri-o://214c97d0252d83475d862c5754d9bb61fd24af69de4ed0a687b8f1d03f1f460f" gracePeriod=30 Jul 07 16:08:31.795904 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:31.795874 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:08:31.906069 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:31.905969 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:08:32.152720 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:32.152690 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-derive-af112f23-6c7f76b67c-zkq54"] Jul 07 16:08:32.152945 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:32.152914 2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-derive-af112f23-6c7f76b67c-zkq54" podUID="691e9fb7-eaa8-46a2-964e-8abe0d190fb4" containerName="maas-api" containerID="cri-o://77c4bb8565e2010b112c51b551fdbc1a4d154109d40aa36776faf7fd2baacb9e" gracePeriod=30 Jul 07 16:08:32.157822 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:32.157762 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-derive-af112f23-6c7f76b67c-zkq54" Jul 07 16:08:32.201790 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:32.201761 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:08:32.398737 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:32.398707 2565 generic.go:356] "Generic (PLEG): container finished" podID="691e9fb7-eaa8-46a2-964e-8abe0d190fb4" containerID="77c4bb8565e2010b112c51b551fdbc1a4d154109d40aa36776faf7fd2baacb9e" exitCode=0 Jul 07 16:08:32.398900 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:32.398775 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-derive-af112f23-6c7f76b67c-zkq54" event={"ID":"691e9fb7-eaa8-46a2-964e-8abe0d190fb4","Type":"ContainerDied","Data":"77c4bb8565e2010b112c51b551fdbc1a4d154109d40aa36776faf7fd2baacb9e"} Jul 07 16:08:32.445795 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:32.445772 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-derive-af112f23-6c7f76b67c-zkq54" Jul 07 16:08:32.557075 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:32.557042 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/691e9fb7-eaa8-46a2-964e-8abe0d190fb4-openshift-service-ca\") pod \"691e9fb7-eaa8-46a2-964e-8abe0d190fb4\" (UID: \"691e9fb7-eaa8-46a2-964e-8abe0d190fb4\") " Jul 07 16:08:32.557276 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:32.557097 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/691e9fb7-eaa8-46a2-964e-8abe0d190fb4-maas-api-tls\") pod \"691e9fb7-eaa8-46a2-964e-8abe0d190fb4\" (UID: \"691e9fb7-eaa8-46a2-964e-8abe0d190fb4\") " Jul 07 16:08:32.557276 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:32.557124 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9sn5g\" (UniqueName: \"kubernetes.io/projected/691e9fb7-eaa8-46a2-964e-8abe0d190fb4-kube-api-access-9sn5g\") pod \"691e9fb7-eaa8-46a2-964e-8abe0d190fb4\" (UID: \"691e9fb7-eaa8-46a2-964e-8abe0d190fb4\") " Jul 07 16:08:32.557460 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:32.557436 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/691e9fb7-eaa8-46a2-964e-8abe0d190fb4-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "691e9fb7-eaa8-46a2-964e-8abe0d190fb4" (UID: "691e9fb7-eaa8-46a2-964e-8abe0d190fb4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 16:08:32.567515 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:32.559329 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/691e9fb7-eaa8-46a2-964e-8abe0d190fb4-kube-api-access-9sn5g" (OuterVolumeSpecName: "kube-api-access-9sn5g") pod "691e9fb7-eaa8-46a2-964e-8abe0d190fb4" (UID: "691e9fb7-eaa8-46a2-964e-8abe0d190fb4"). InnerVolumeSpecName "kube-api-access-9sn5g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 16:08:32.584515 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:32.583988 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/691e9fb7-eaa8-46a2-964e-8abe0d190fb4-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "691e9fb7-eaa8-46a2-964e-8abe0d190fb4" (UID: "691e9fb7-eaa8-46a2-964e-8abe0d190fb4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 16:08:32.658715 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:32.658672 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9sn5g\" (UniqueName: \"kubernetes.io/projected/691e9fb7-eaa8-46a2-964e-8abe0d190fb4-kube-api-access-9sn5g\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 07 16:08:32.658715 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:32.658715 2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/691e9fb7-eaa8-46a2-964e-8abe0d190fb4-openshift-service-ca\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 07 16:08:32.659264 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:32.658731 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/691e9fb7-eaa8-46a2-964e-8abe0d190fb4-maas-api-tls\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 07 16:08:33.406263 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:33.406233 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-derive-af112f23-6c7f76b67c-zkq54" event={"ID":"691e9fb7-eaa8-46a2-964e-8abe0d190fb4","Type":"ContainerDied","Data":"f4dbbd99f2264c4f1ab847e0ae064877bd3c9b822b4f321496dca0085ed39cf2"} Jul 07 16:08:33.406263 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:33.406250 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-derive-af112f23-6c7f76b67c-zkq54" Jul 07 16:08:33.406465 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:33.406294 2565 scope.go:117] "RemoveContainer" containerID="77c4bb8565e2010b112c51b551fdbc1a4d154109d40aa36776faf7fd2baacb9e" Jul 07 16:08:33.433438 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:33.433405 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-derive-af112f23-6c7f76b67c-zkq54"] Jul 07 16:08:33.435167 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:33.435142 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:08:33.438753 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:33.438732 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-derive-af112f23-6c7f76b67c-zkq54"] Jul 07 16:08:33.663768 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:33.663673 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="691e9fb7-eaa8-46a2-964e-8abe0d190fb4" path="/var/lib/kubelet/pods/691e9fb7-eaa8-46a2-964e-8abe0d190fb4/volumes" Jul 07 16:08:38.834451 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:38.834422 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-af112f23-openshift-default-ccb75999-zcd8d" Jul 07 16:08:38.925300 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:38.925208 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75be73ad-40d7-4a58-b36d-8ababf74ccc6-workload-socket\") pod \"75be73ad-40d7-4a58-b36d-8ababf74ccc6\" (UID: \"75be73ad-40d7-4a58-b36d-8ababf74ccc6\") " Jul 07 16:08:38.925300 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:38.925248 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75be73ad-40d7-4a58-b36d-8ababf74ccc6-istio-podinfo\") pod \"75be73ad-40d7-4a58-b36d-8ababf74ccc6\" (UID: \"75be73ad-40d7-4a58-b36d-8ababf74ccc6\") " Jul 07 16:08:38.925300 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:38.925292 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75be73ad-40d7-4a58-b36d-8ababf74ccc6-istio-token\") pod \"75be73ad-40d7-4a58-b36d-8ababf74ccc6\" (UID: \"75be73ad-40d7-4a58-b36d-8ababf74ccc6\") " Jul 07 16:08:38.925601 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:38.925314 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75be73ad-40d7-4a58-b36d-8ababf74ccc6-credential-socket\") pod \"75be73ad-40d7-4a58-b36d-8ababf74ccc6\" (UID: \"75be73ad-40d7-4a58-b36d-8ababf74ccc6\") " Jul 07 16:08:38.925601 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:38.925345 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75be73ad-40d7-4a58-b36d-8ababf74ccc6-istio-envoy\") pod \"75be73ad-40d7-4a58-b36d-8ababf74ccc6\" (UID: \"75be73ad-40d7-4a58-b36d-8ababf74ccc6\") " Jul 07 16:08:38.925601 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:38.925394 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75be73ad-40d7-4a58-b36d-8ababf74ccc6-istiod-ca-cert\") pod \"75be73ad-40d7-4a58-b36d-8ababf74ccc6\" (UID: \"75be73ad-40d7-4a58-b36d-8ababf74ccc6\") " Jul 07 16:08:38.925601 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:38.925412 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75be73ad-40d7-4a58-b36d-8ababf74ccc6-workload-certs\") pod \"75be73ad-40d7-4a58-b36d-8ababf74ccc6\" (UID: \"75be73ad-40d7-4a58-b36d-8ababf74ccc6\") " Jul 07 16:08:38.925601 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:38.925474 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8m2wb\" (UniqueName: \"kubernetes.io/projected/75be73ad-40d7-4a58-b36d-8ababf74ccc6-kube-api-access-8m2wb\") pod \"75be73ad-40d7-4a58-b36d-8ababf74ccc6\" (UID: \"75be73ad-40d7-4a58-b36d-8ababf74ccc6\") " Jul 07 16:08:38.925601 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:38.925523 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75be73ad-40d7-4a58-b36d-8ababf74ccc6-istio-data\") pod \"75be73ad-40d7-4a58-b36d-8ababf74ccc6\" (UID: \"75be73ad-40d7-4a58-b36d-8ababf74ccc6\") " Jul 07 16:08:38.925601 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:38.925555 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75be73ad-40d7-4a58-b36d-8ababf74ccc6-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "75be73ad-40d7-4a58-b36d-8ababf74ccc6" (UID: "75be73ad-40d7-4a58-b36d-8ababf74ccc6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 16:08:38.925948 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:38.925679 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75be73ad-40d7-4a58-b36d-8ababf74ccc6-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "75be73ad-40d7-4a58-b36d-8ababf74ccc6" (UID: "75be73ad-40d7-4a58-b36d-8ababf74ccc6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 16:08:38.925948 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:38.925862 2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75be73ad-40d7-4a58-b36d-8ababf74ccc6-credential-socket\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 07 16:08:38.925948 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:38.925882 2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75be73ad-40d7-4a58-b36d-8ababf74ccc6-workload-socket\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 07 16:08:38.925948 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:38.925875 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75be73ad-40d7-4a58-b36d-8ababf74ccc6-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "75be73ad-40d7-4a58-b36d-8ababf74ccc6" (UID: "75be73ad-40d7-4a58-b36d-8ababf74ccc6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 16:08:38.926132 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:38.926084 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75be73ad-40d7-4a58-b36d-8ababf74ccc6-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "75be73ad-40d7-4a58-b36d-8ababf74ccc6" (UID: "75be73ad-40d7-4a58-b36d-8ababf74ccc6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 16:08:38.926824 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:38.926794 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75be73ad-40d7-4a58-b36d-8ababf74ccc6-istio-data" (OuterVolumeSpecName: "istio-data") pod "75be73ad-40d7-4a58-b36d-8ababf74ccc6" (UID: "75be73ad-40d7-4a58-b36d-8ababf74ccc6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 16:08:38.927608 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:38.927584 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/75be73ad-40d7-4a58-b36d-8ababf74ccc6-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "75be73ad-40d7-4a58-b36d-8ababf74ccc6" (UID: "75be73ad-40d7-4a58-b36d-8ababf74ccc6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 07 16:08:38.927693 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:38.927600 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75be73ad-40d7-4a58-b36d-8ababf74ccc6-istio-token" (OuterVolumeSpecName: "istio-token") pod "75be73ad-40d7-4a58-b36d-8ababf74ccc6" (UID: "75be73ad-40d7-4a58-b36d-8ababf74ccc6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 16:08:38.927693 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:38.927617 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75be73ad-40d7-4a58-b36d-8ababf74ccc6-kube-api-access-8m2wb" (OuterVolumeSpecName: "kube-api-access-8m2wb") pod "75be73ad-40d7-4a58-b36d-8ababf74ccc6" (UID: "75be73ad-40d7-4a58-b36d-8ababf74ccc6"). InnerVolumeSpecName "kube-api-access-8m2wb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 16:08:38.927800 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:38.927741 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75be73ad-40d7-4a58-b36d-8ababf74ccc6-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "75be73ad-40d7-4a58-b36d-8ababf74ccc6" (UID: "75be73ad-40d7-4a58-b36d-8ababf74ccc6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 16:08:39.026583 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:39.026542 2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75be73ad-40d7-4a58-b36d-8ababf74ccc6-istiod-ca-cert\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 07 16:08:39.026583 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:39.026576 2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75be73ad-40d7-4a58-b36d-8ababf74ccc6-workload-certs\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 07 16:08:39.026583 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:39.026586 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8m2wb\" (UniqueName: \"kubernetes.io/projected/75be73ad-40d7-4a58-b36d-8ababf74ccc6-kube-api-access-8m2wb\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 07 16:08:39.026884 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:39.026598 2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75be73ad-40d7-4a58-b36d-8ababf74ccc6-istio-data\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 07 16:08:39.026884 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:39.026612 2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75be73ad-40d7-4a58-b36d-8ababf74ccc6-istio-podinfo\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 07 16:08:39.026884 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:39.026624 2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75be73ad-40d7-4a58-b36d-8ababf74ccc6-istio-token\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 07 16:08:39.026884 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:39.026631 2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75be73ad-40d7-4a58-b36d-8ababf74ccc6-istio-envoy\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 07 16:08:39.185235 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:39.185141 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:08:39.260372 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:39.260324 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:08:39.438478 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:39.438380 2565 generic.go:356] "Generic (PLEG): container finished" podID="75be73ad-40d7-4a58-b36d-8ababf74ccc6" containerID="214c97d0252d83475d862c5754d9bb61fd24af69de4ed0a687b8f1d03f1f460f" exitCode=0 Jul 07 16:08:39.438478 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:39.438427 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-af112f23-openshift-default-ccb75999-zcd8d" Jul 07 16:08:39.438478 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:39.438465 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-af112f23-openshift-default-ccb75999-zcd8d" event={"ID":"75be73ad-40d7-4a58-b36d-8ababf74ccc6","Type":"ContainerDied","Data":"214c97d0252d83475d862c5754d9bb61fd24af69de4ed0a687b8f1d03f1f460f"} Jul 07 16:08:39.438775 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:39.438541 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-af112f23-openshift-default-ccb75999-zcd8d" event={"ID":"75be73ad-40d7-4a58-b36d-8ababf74ccc6","Type":"ContainerDied","Data":"c4c99296bec43aa7d2e42dbde8527f04a2f4bf16dbbf0fed9bcc0d22e2d9c681"} Jul 07 16:08:39.438775 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:39.438567 2565 scope.go:117] "RemoveContainer" containerID="214c97d0252d83475d862c5754d9bb61fd24af69de4ed0a687b8f1d03f1f460f" Jul 07 16:08:39.448516 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:39.448482 2565 scope.go:117] "RemoveContainer" containerID="214c97d0252d83475d862c5754d9bb61fd24af69de4ed0a687b8f1d03f1f460f" Jul 07 16:08:39.448755 ip-10-0-132-65 kubenswrapper[2565]: E0707 16:08:39.448737 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"214c97d0252d83475d862c5754d9bb61fd24af69de4ed0a687b8f1d03f1f460f\": container with ID starting with 214c97d0252d83475d862c5754d9bb61fd24af69de4ed0a687b8f1d03f1f460f not found: ID does not exist" containerID="214c97d0252d83475d862c5754d9bb61fd24af69de4ed0a687b8f1d03f1f460f" Jul 07 16:08:39.448815 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:39.448764 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"214c97d0252d83475d862c5754d9bb61fd24af69de4ed0a687b8f1d03f1f460f"} err="failed to get container status \"214c97d0252d83475d862c5754d9bb61fd24af69de4ed0a687b8f1d03f1f460f\": rpc error: code = NotFound desc = could not find container \"214c97d0252d83475d862c5754d9bb61fd24af69de4ed0a687b8f1d03f1f460f\": container with ID starting with 214c97d0252d83475d862c5754d9bb61fd24af69de4ed0a687b8f1d03f1f460f not found: ID does not exist" Jul 07 16:08:39.466518 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:39.466473 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-af112f23-openshift-default-ccb75999-zcd8d"] Jul 07 16:08:39.469116 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:39.469087 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-af112f23-openshift-default-ccb75999-zcd8d"] Jul 07 16:08:39.661196 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:39.661161 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="75be73ad-40d7-4a58-b36d-8ababf74ccc6" path="/var/lib/kubelet/pods/75be73ad-40d7-4a58-b36d-8ababf74ccc6/volumes" Jul 07 16:08:39.696768 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:39.696676 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:08:39.791006 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:39.790973 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:08:39.890542 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:39.890487 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:08:41.411740 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:41.411708 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:08:48.370705 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:48.370669 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:08:48.443937 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:48.443905 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:08:48.535202 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:48.535166 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:08:48.635005 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:48.634918 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:08:48.735753 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:48.735706 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:08:49.154675 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:08:49.154639 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:09:15.141299 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:09:15.141260 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:09:21.541009 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:09:21.540972 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:09:21.643639 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:09:21.643610 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:09:21.741529 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:09:21.741484 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:09:37.673016 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:09:37.672932 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:09:44.745187 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:09:44.745107 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:09:44.817647 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:09:44.817614 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:09:44.919998 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:09:44.919964 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:09:45.015515 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:09:45.015401 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:09:55.543784 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:09:55.543749 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:09:56.349195 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:09:56.349150 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:10:03.093164 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:10:03.093121 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:10:03.203713 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:10:03.203680 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:10:03.388937 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:10:03.388852 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:10:03.500304 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:10:03.500266 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:10:04.309807 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:10:04.309773 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:10:04.414007 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:10:04.413975 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:10:06.318318 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:10:06.318285 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:10:13.016809 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:10:13.016774 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:10:13.114697 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:10:13.114659 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:10:13.709301 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:10:13.709267 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:10:15.257338 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:10:15.257251 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:10:15.325652 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:10:15.325618 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:10:20.395184 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:10:20.395151 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:10:20.478180 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:10:20.478148 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:10:20.646849 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:10:20.646760 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:10:20.968187 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:10:20.968110 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-cf3310ce-openshift-default-94dd57675-njq8z"] Jul 07 16:10:20.968533 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:10:20.968521 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="75be73ad-40d7-4a58-b36d-8ababf74ccc6" containerName="istio-proxy" Jul 07 16:10:20.968590 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:10:20.968534 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="75be73ad-40d7-4a58-b36d-8ababf74ccc6" containerName="istio-proxy" Jul 07 16:10:20.968590 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:10:20.968544 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="691e9fb7-eaa8-46a2-964e-8abe0d190fb4" containerName="maas-api" Jul 07 16:10:20.968590 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:10:20.968550 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="691e9fb7-eaa8-46a2-964e-8abe0d190fb4" containerName="maas-api" Jul 07 16:10:20.968688 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:10:20.968609 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="75be73ad-40d7-4a58-b36d-8ababf74ccc6" containerName="istio-proxy" Jul 07 16:10:20.968688 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:10:20.968621 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="691e9fb7-eaa8-46a2-964e-8abe0d190fb4" containerName="maas-api" Jul 07 16:10:20.970722 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:10:20.970706 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-cf3310ce-openshift-default-94dd57675-njq8z" Jul 07 16:10:20.975263 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:10:20.975243 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-cf3310ce-openshift-default-dockercfg-vv5ht\"" Jul 07 16:10:21.002081 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:10:21.002049 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-cf3310ce-openshift-default-94dd57675-njq8z"] Jul 07 16:10:21.003349 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:10:21.003322 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:10:21.061608 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:10:21.061558 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34fd2ca7-0000-4409-ad58-9253a81c5ba4-istio-token\") pod \"e2e-mt-cf3310ce-openshift-default-94dd57675-njq8z\" (UID: \"34fd2ca7-0000-4409-ad58-9253a81c5ba4\") " pod="openshift-ingress/e2e-mt-cf3310ce-openshift-default-94dd57675-njq8z" Jul 07 16:10:21.061800 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:10:21.061615 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34fd2ca7-0000-4409-ad58-9253a81c5ba4-workload-certs\") pod \"e2e-mt-cf3310ce-openshift-default-94dd57675-njq8z\" (UID: \"34fd2ca7-0000-4409-ad58-9253a81c5ba4\") " pod="openshift-ingress/e2e-mt-cf3310ce-openshift-default-94dd57675-njq8z" Jul 07 16:10:21.061800 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:10:21.061646 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7swfd\" (UniqueName: \"kubernetes.io/projected/34fd2ca7-0000-4409-ad58-9253a81c5ba4-kube-api-access-7swfd\") pod \"e2e-mt-cf3310ce-openshift-default-94dd57675-njq8z\" (UID: \"34fd2ca7-0000-4409-ad58-9253a81c5ba4\") " pod="openshift-ingress/e2e-mt-cf3310ce-openshift-default-94dd57675-njq8z" Jul 07 16:10:21.061800 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:10:21.061676 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34fd2ca7-0000-4409-ad58-9253a81c5ba4-istiod-ca-cert\") pod \"e2e-mt-cf3310ce-openshift-default-94dd57675-njq8z\" (UID: \"34fd2ca7-0000-4409-ad58-9253a81c5ba4\") " pod="openshift-ingress/e2e-mt-cf3310ce-openshift-default-94dd57675-njq8z" Jul 07 16:10:21.061800 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:10:21.061721 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34fd2ca7-0000-4409-ad58-9253a81c5ba4-istio-podinfo\") pod \"e2e-mt-cf3310ce-openshift-default-94dd57675-njq8z\" (UID: \"34fd2ca7-0000-4409-ad58-9253a81c5ba4\") " pod="openshift-ingress/e2e-mt-cf3310ce-openshift-default-94dd57675-njq8z" Jul 07 16:10:21.061800 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:10:21.061748 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34fd2ca7-0000-4409-ad58-9253a81c5ba4-credential-socket\") pod \"e2e-mt-cf3310ce-openshift-default-94dd57675-njq8z\" (UID: \"34fd2ca7-0000-4409-ad58-9253a81c5ba4\") " pod="openshift-ingress/e2e-mt-cf3310ce-openshift-default-94dd57675-njq8z" Jul 07 16:10:21.062069 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:10:21.061825 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34fd2ca7-0000-4409-ad58-9253a81c5ba4-istio-envoy\") pod \"e2e-mt-cf3310ce-openshift-default-94dd57675-njq8z\" (UID: \"34fd2ca7-0000-4409-ad58-9253a81c5ba4\") " pod="openshift-ingress/e2e-mt-cf3310ce-openshift-default-94dd57675-njq8z" Jul 07 16:10:21.062069 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:10:21.061846 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34fd2ca7-0000-4409-ad58-9253a81c5ba4-istio-data\") pod \"e2e-mt-cf3310ce-openshift-default-94dd57675-njq8z\" (UID: \"34fd2ca7-0000-4409-ad58-9253a81c5ba4\") " pod="openshift-ingress/e2e-mt-cf3310ce-openshift-default-94dd57675-njq8z" Jul 07 16:10:21.062069 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:10:21.061864 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34fd2ca7-0000-4409-ad58-9253a81c5ba4-workload-socket\") pod \"e2e-mt-cf3310ce-openshift-default-94dd57675-njq8z\" (UID: \"34fd2ca7-0000-4409-ad58-9253a81c5ba4\") " pod="openshift-ingress/e2e-mt-cf3310ce-openshift-default-94dd57675-njq8z" Jul 07 16:10:21.162412 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:10:21.162364 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34fd2ca7-0000-4409-ad58-9253a81c5ba4-istio-envoy\") pod \"e2e-mt-cf3310ce-openshift-default-94dd57675-njq8z\" (UID: \"34fd2ca7-0000-4409-ad58-9253a81c5ba4\") " pod="openshift-ingress/e2e-mt-cf3310ce-openshift-default-94dd57675-njq8z" Jul 07 16:10:21.162412 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:10:21.162401 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34fd2ca7-0000-4409-ad58-9253a81c5ba4-istio-data\") pod \"e2e-mt-cf3310ce-openshift-default-94dd57675-njq8z\" (UID: \"34fd2ca7-0000-4409-ad58-9253a81c5ba4\") " pod="openshift-ingress/e2e-mt-cf3310ce-openshift-default-94dd57675-njq8z" Jul 07 16:10:21.162412 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:10:21.162424 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34fd2ca7-0000-4409-ad58-9253a81c5ba4-workload-socket\") pod \"e2e-mt-cf3310ce-openshift-default-94dd57675-njq8z\" (UID: \"34fd2ca7-0000-4409-ad58-9253a81c5ba4\") " pod="openshift-ingress/e2e-mt-cf3310ce-openshift-default-94dd57675-njq8z" Jul 07 16:10:21.162745 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:10:21.162505 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34fd2ca7-0000-4409-ad58-9253a81c5ba4-istio-token\") pod \"e2e-mt-cf3310ce-openshift-default-94dd57675-njq8z\" (UID: \"34fd2ca7-0000-4409-ad58-9253a81c5ba4\") " pod="openshift-ingress/e2e-mt-cf3310ce-openshift-default-94dd57675-njq8z" Jul 07 16:10:21.162745 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:10:21.162530 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34fd2ca7-0000-4409-ad58-9253a81c5ba4-workload-certs\") pod \"e2e-mt-cf3310ce-openshift-default-94dd57675-njq8z\" (UID: \"34fd2ca7-0000-4409-ad58-9253a81c5ba4\") " pod="openshift-ingress/e2e-mt-cf3310ce-openshift-default-94dd57675-njq8z" Jul 07 16:10:21.162745 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:10:21.162549 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7swfd\" (UniqueName: \"kubernetes.io/projected/34fd2ca7-0000-4409-ad58-9253a81c5ba4-kube-api-access-7swfd\") pod \"e2e-mt-cf3310ce-openshift-default-94dd57675-njq8z\" (UID: \"34fd2ca7-0000-4409-ad58-9253a81c5ba4\") " pod="openshift-ingress/e2e-mt-cf3310ce-openshift-default-94dd57675-njq8z" Jul 07 16:10:21.162745 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:10:21.162576 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34fd2ca7-0000-4409-ad58-9253a81c5ba4-istiod-ca-cert\") pod \"e2e-mt-cf3310ce-openshift-default-94dd57675-njq8z\" (UID: \"34fd2ca7-0000-4409-ad58-9253a81c5ba4\") " pod="openshift-ingress/e2e-mt-cf3310ce-openshift-default-94dd57675-njq8z" Jul 07 16:10:21.162954 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:10:21.162844 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34fd2ca7-0000-4409-ad58-9253a81c5ba4-istio-data\") pod \"e2e-mt-cf3310ce-openshift-default-94dd57675-njq8z\" (UID: \"34fd2ca7-0000-4409-ad58-9253a81c5ba4\") " pod="openshift-ingress/e2e-mt-cf3310ce-openshift-default-94dd57675-njq8z" Jul 07 16:10:21.162954 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:10:21.162866 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34fd2ca7-0000-4409-ad58-9253a81c5ba4-workload-socket\") pod \"e2e-mt-cf3310ce-openshift-default-94dd57675-njq8z\" (UID: \"34fd2ca7-0000-4409-ad58-9253a81c5ba4\") " pod="openshift-ingress/e2e-mt-cf3310ce-openshift-default-94dd57675-njq8z" Jul 07 16:10:21.163064 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:10:21.163030 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34fd2ca7-0000-4409-ad58-9253a81c5ba4-workload-certs\") pod \"e2e-mt-cf3310ce-openshift-default-94dd57675-njq8z\" (UID: \"34fd2ca7-0000-4409-ad58-9253a81c5ba4\") " pod="openshift-ingress/e2e-mt-cf3310ce-openshift-default-94dd57675-njq8z" Jul 07 16:10:21.163118 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:10:21.163097 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34fd2ca7-0000-4409-ad58-9253a81c5ba4-istio-podinfo\") pod \"e2e-mt-cf3310ce-openshift-default-94dd57675-njq8z\" (UID: \"34fd2ca7-0000-4409-ad58-9253a81c5ba4\") " pod="openshift-ingress/e2e-mt-cf3310ce-openshift-default-94dd57675-njq8z" Jul 07 16:10:21.163175 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:10:21.163138 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34fd2ca7-0000-4409-ad58-9253a81c5ba4-credential-socket\") pod \"e2e-mt-cf3310ce-openshift-default-94dd57675-njq8z\" (UID: \"34fd2ca7-0000-4409-ad58-9253a81c5ba4\") " pod="openshift-ingress/e2e-mt-cf3310ce-openshift-default-94dd57675-njq8z" Jul 07 16:10:21.163286 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:10:21.163266 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34fd2ca7-0000-4409-ad58-9253a81c5ba4-istiod-ca-cert\") pod \"e2e-mt-cf3310ce-openshift-default-94dd57675-njq8z\" (UID: \"34fd2ca7-0000-4409-ad58-9253a81c5ba4\") " pod="openshift-ingress/e2e-mt-cf3310ce-openshift-default-94dd57675-njq8z" Jul 07 16:10:21.163409 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:10:21.163390 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34fd2ca7-0000-4409-ad58-9253a81c5ba4-credential-socket\") pod \"e2e-mt-cf3310ce-openshift-default-94dd57675-njq8z\" (UID: \"34fd2ca7-0000-4409-ad58-9253a81c5ba4\") " pod="openshift-ingress/e2e-mt-cf3310ce-openshift-default-94dd57675-njq8z" Jul 07 16:10:21.164776 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:10:21.164751 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34fd2ca7-0000-4409-ad58-9253a81c5ba4-istio-envoy\") pod \"e2e-mt-cf3310ce-openshift-default-94dd57675-njq8z\" (UID: \"34fd2ca7-0000-4409-ad58-9253a81c5ba4\") " pod="openshift-ingress/e2e-mt-cf3310ce-openshift-default-94dd57675-njq8z" Jul 07 16:10:21.165096 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:10:21.165079 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34fd2ca7-0000-4409-ad58-9253a81c5ba4-istio-podinfo\") pod \"e2e-mt-cf3310ce-openshift-default-94dd57675-njq8z\" (UID: \"34fd2ca7-0000-4409-ad58-9253a81c5ba4\") " pod="openshift-ingress/e2e-mt-cf3310ce-openshift-default-94dd57675-njq8z" Jul 07 16:10:21.175371 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:10:21.175342 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34fd2ca7-0000-4409-ad58-9253a81c5ba4-istio-token\") pod \"e2e-mt-cf3310ce-openshift-default-94dd57675-njq8z\" (UID: \"34fd2ca7-0000-4409-ad58-9253a81c5ba4\") " pod="openshift-ingress/e2e-mt-cf3310ce-openshift-default-94dd57675-njq8z" Jul 07 16:10:21.176011 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:10:21.175990 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7swfd\" (UniqueName: \"kubernetes.io/projected/34fd2ca7-0000-4409-ad58-9253a81c5ba4-kube-api-access-7swfd\") pod \"e2e-mt-cf3310ce-openshift-default-94dd57675-njq8z\" (UID: \"34fd2ca7-0000-4409-ad58-9253a81c5ba4\") " pod="openshift-ingress/e2e-mt-cf3310ce-openshift-default-94dd57675-njq8z" Jul 07 16:10:21.281384 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:10:21.281299 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-cf3310ce-openshift-default-94dd57675-njq8z" Jul 07 16:10:21.313666 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:10:21.313636 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:10:21.433543 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:10:21.433425 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-cf3310ce-openshift-default-94dd57675-njq8z"] Jul 07 16:10:21.435834 ip-10-0-132-65 kubenswrapper[2565]: W0707 16:10:21.435809 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34fd2ca7_0000_4409_ad58_9253a81c5ba4.slice/crio-a3ecbd3d311209da2edddcc881cb3e920e2996e97a702b038c5e5f45c7ca7839 WatchSource:0}: Error finding container a3ecbd3d311209da2edddcc881cb3e920e2996e97a702b038c5e5f45c7ca7839: Status 404 returned error can't find the container with id a3ecbd3d311209da2edddcc881cb3e920e2996e97a702b038c5e5f45c7ca7839 Jul 07 16:10:21.437886 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:10:21.437854 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 07 16:10:21.437954 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:10:21.437915 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 07 16:10:21.437954 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:10:21.437941 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 07 16:10:21.940524 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:10:21.940475 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-cf3310ce-openshift-default-94dd57675-njq8z" event={"ID":"34fd2ca7-0000-4409-ad58-9253a81c5ba4","Type":"ContainerStarted","Data":"1c193ecf4b390590cea76a91b7c920cd4a0b7eb2ba89f1be8a89e1677bd53f91"} Jul 07 16:10:21.940736 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:10:21.940549 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-cf3310ce-openshift-default-94dd57675-njq8z" event={"ID":"34fd2ca7-0000-4409-ad58-9253a81c5ba4","Type":"ContainerStarted","Data":"a3ecbd3d311209da2edddcc881cb3e920e2996e97a702b038c5e5f45c7ca7839"} Jul 07 16:10:21.969293 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:10:21.969241 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-cf3310ce-openshift-default-94dd57675-njq8z" podStartSLOduration=1.969227064 podStartE2EDuration="1.969227064s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:10:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:10:21.967521295 +0000 UTC m=+1760.924098765" watchObservedRunningTime="2026-07-07 16:10:21.969227064 +0000 UTC m=+1760.925804530" Jul 07 16:10:22.282229 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:10:22.282139 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-cf3310ce-openshift-default-94dd57675-njq8z" Jul 07 16:10:22.287221 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:10:22.287195 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-cf3310ce-openshift-default-94dd57675-njq8z" Jul 07 16:10:22.946042 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:10:22.946009 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-cf3310ce-openshift-default-94dd57675-njq8z" Jul 07 16:10:22.947003 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:10:22.946984 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-cf3310ce-openshift-default-94dd57675-njq8z" Jul 07 16:10:24.124699 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:10:24.124659 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-mt-cf3310ce-7d5d4f87ff-c7cx9"] Jul 07 16:10:24.127581 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:10:24.127559 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-cf3310ce-7d5d4f87ff-c7cx9" Jul 07 16:10:24.130434 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:10:24.130412 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-dockercfg-9bqq6\"" Jul 07 16:10:24.130741 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:10:24.130725 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-mt-cf3310ce-serving-cert\"" Jul 07 16:10:24.141918 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:10:24.141891 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:10:24.142778 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:10:24.142749 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-mt-cf3310ce-7d5d4f87ff-c7cx9"] Jul 07 16:10:24.224100 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:10:24.224071 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:10:24.294230 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:10:24.294196 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4bdd\" (UniqueName: \"kubernetes.io/projected/c2d0b044-0bac-4781-a752-1ec6a575208e-kube-api-access-k4bdd\") pod \"maas-api-e2e-mt-cf3310ce-7d5d4f87ff-c7cx9\" (UID: \"c2d0b044-0bac-4781-a752-1ec6a575208e\") " pod="opendatahub/maas-api-e2e-mt-cf3310ce-7d5d4f87ff-c7cx9" Jul 07 16:10:24.294230 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:10:24.294233 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2d0b044-0bac-4781-a752-1ec6a575208e-openshift-service-ca\") pod \"maas-api-e2e-mt-cf3310ce-7d5d4f87ff-c7cx9\" (UID: \"c2d0b044-0bac-4781-a752-1ec6a575208e\") " pod="opendatahub/maas-api-e2e-mt-cf3310ce-7d5d4f87ff-c7cx9" Jul 07 16:10:24.294458 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:10:24.294257 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c2d0b044-0bac-4781-a752-1ec6a575208e-maas-api-tls\") pod \"maas-api-e2e-mt-cf3310ce-7d5d4f87ff-c7cx9\" (UID: \"c2d0b044-0bac-4781-a752-1ec6a575208e\") " pod="opendatahub/maas-api-e2e-mt-cf3310ce-7d5d4f87ff-c7cx9" Jul 07 16:10:24.339279 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:10:24.339245 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:10:24.395165 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:10:24.395054 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4bdd\" (UniqueName: \"kubernetes.io/projected/c2d0b044-0bac-4781-a752-1ec6a575208e-kube-api-access-k4bdd\") pod \"maas-api-e2e-mt-cf3310ce-7d5d4f87ff-c7cx9\" (UID: \"c2d0b044-0bac-4781-a752-1ec6a575208e\") " pod="opendatahub/maas-api-e2e-mt-cf3310ce-7d5d4f87ff-c7cx9" Jul 07 16:10:24.395165 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:10:24.395108 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2d0b044-0bac-4781-a752-1ec6a575208e-openshift-service-ca\") pod \"maas-api-e2e-mt-cf3310ce-7d5d4f87ff-c7cx9\" (UID: \"c2d0b044-0bac-4781-a752-1ec6a575208e\") " pod="opendatahub/maas-api-e2e-mt-cf3310ce-7d5d4f87ff-c7cx9" Jul 07 16:10:24.395165 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:10:24.395137 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c2d0b044-0bac-4781-a752-1ec6a575208e-maas-api-tls\") pod \"maas-api-e2e-mt-cf3310ce-7d5d4f87ff-c7cx9\" (UID: \"c2d0b044-0bac-4781-a752-1ec6a575208e\") " pod="opendatahub/maas-api-e2e-mt-cf3310ce-7d5d4f87ff-c7cx9" Jul 07 16:10:24.395898 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:10:24.395877 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2d0b044-0bac-4781-a752-1ec6a575208e-openshift-service-ca\") pod \"maas-api-e2e-mt-cf3310ce-7d5d4f87ff-c7cx9\" (UID: \"c2d0b044-0bac-4781-a752-1ec6a575208e\") " pod="opendatahub/maas-api-e2e-mt-cf3310ce-7d5d4f87ff-c7cx9" Jul 07 16:10:24.397607 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:10:24.397587 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c2d0b044-0bac-4781-a752-1ec6a575208e-maas-api-tls\") pod \"maas-api-e2e-mt-cf3310ce-7d5d4f87ff-c7cx9\" (UID: \"c2d0b044-0bac-4781-a752-1ec6a575208e\") " pod="opendatahub/maas-api-e2e-mt-cf3310ce-7d5d4f87ff-c7cx9" Jul 07 16:10:24.425662 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:10:24.425633 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4bdd\" (UniqueName: \"kubernetes.io/projected/c2d0b044-0bac-4781-a752-1ec6a575208e-kube-api-access-k4bdd\") pod \"maas-api-e2e-mt-cf3310ce-7d5d4f87ff-c7cx9\" (UID: \"c2d0b044-0bac-4781-a752-1ec6a575208e\") " pod="opendatahub/maas-api-e2e-mt-cf3310ce-7d5d4f87ff-c7cx9" Jul 07 16:10:24.438732 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:10:24.438704 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-cf3310ce-7d5d4f87ff-c7cx9" Jul 07 16:10:24.600664 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:10:24.600631 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-mt-cf3310ce-7d5d4f87ff-c7cx9"] Jul 07 16:10:24.601653 ip-10-0-132-65 kubenswrapper[2565]: W0707 16:10:24.601619 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2d0b044_0bac_4781_a752_1ec6a575208e.slice/crio-933ab327384b6650be34ea3864acb30eb0e9fe1bb37dbb590e7b773b8b494224 WatchSource:0}: Error finding container 933ab327384b6650be34ea3864acb30eb0e9fe1bb37dbb590e7b773b8b494224: Status 404 returned error can't find the container with id 933ab327384b6650be34ea3864acb30eb0e9fe1bb37dbb590e7b773b8b494224 Jul 07 16:10:24.832850 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:10:24.832814 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:10:24.935086 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:10:24.935043 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:10:24.956974 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:10:24.956942 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-cf3310ce-7d5d4f87ff-c7cx9" event={"ID":"c2d0b044-0bac-4781-a752-1ec6a575208e","Type":"ContainerStarted","Data":"933ab327384b6650be34ea3864acb30eb0e9fe1bb37dbb590e7b773b8b494224"} Jul 07 16:10:25.963349 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:10:25.963312 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-cf3310ce-7d5d4f87ff-c7cx9" event={"ID":"c2d0b044-0bac-4781-a752-1ec6a575208e","Type":"ContainerStarted","Data":"2e2648e0e320749eb2f2fcb848f1bd4b803434e5715cf534f53dc93093ea5e28"} Jul 07 16:10:25.963745 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:10:25.963422 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-mt-cf3310ce-7d5d4f87ff-c7cx9" Jul 07 16:10:25.986955 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:10:25.986907 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-mt-cf3310ce-7d5d4f87ff-c7cx9" podStartSLOduration=1.3063035410000001 podStartE2EDuration="1.986891818s" totalImagesPullingTime="680.588277ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:10:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:10:25.985106687 +0000 UTC m=+1764.941684155" watchObservedRunningTime="2026-07-07 16:10:25.986891818 +0000 UTC m=+1764.943469285" Jul 07 16:10:28.561094 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:10:28.561065 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:10:29.316675 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:10:29.316637 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:10:29.412222 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:10:29.412187 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:10:29.827690 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:10:29.827664 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:10:31.980983 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:10:31.980946 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-mt-cf3310ce-7d5d4f87ff-c7cx9" Jul 07 16:10:36.556467 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:10:36.556434 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:10:36.660162 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:10:36.660127 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:10:37.026586 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:10:37.026551 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:10:37.429887 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:10:37.429813 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:10:37.456011 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:10:37.455980 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-mt-cf3310ce-7d5d4f87ff-c7cx9"] Jul 07 16:10:37.456223 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:10:37.456200 2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-mt-cf3310ce-7d5d4f87ff-c7cx9" podUID="c2d0b044-0bac-4781-a752-1ec6a575208e" containerName="maas-api" containerID="cri-o://2e2648e0e320749eb2f2fcb848f1bd4b803434e5715cf534f53dc93093ea5e28" gracePeriod=30 Jul 07 16:10:37.531277 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:10:37.531249 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:10:37.715779 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:10:37.715754 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-cf3310ce-7d5d4f87ff-c7cx9" Jul 07 16:10:37.818703 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:10:37.818671 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2d0b044-0bac-4781-a752-1ec6a575208e-openshift-service-ca\") pod \"c2d0b044-0bac-4781-a752-1ec6a575208e\" (UID: \"c2d0b044-0bac-4781-a752-1ec6a575208e\") " Jul 07 16:10:37.818881 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:10:37.818772 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k4bdd\" (UniqueName: \"kubernetes.io/projected/c2d0b044-0bac-4781-a752-1ec6a575208e-kube-api-access-k4bdd\") pod \"c2d0b044-0bac-4781-a752-1ec6a575208e\" (UID: \"c2d0b044-0bac-4781-a752-1ec6a575208e\") " Jul 07 16:10:37.818940 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:10:37.818884 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c2d0b044-0bac-4781-a752-1ec6a575208e-maas-api-tls\") pod \"c2d0b044-0bac-4781-a752-1ec6a575208e\" (UID: \"c2d0b044-0bac-4781-a752-1ec6a575208e\") " Jul 07 16:10:37.819072 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:10:37.819050 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2d0b044-0bac-4781-a752-1ec6a575208e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c2d0b044-0bac-4781-a752-1ec6a575208e" (UID: "c2d0b044-0bac-4781-a752-1ec6a575208e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 16:10:37.819308 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:10:37.819288 2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2d0b044-0bac-4781-a752-1ec6a575208e-openshift-service-ca\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 07 16:10:37.820856 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:10:37.820833 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c2d0b044-0bac-4781-a752-1ec6a575208e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c2d0b044-0bac-4781-a752-1ec6a575208e" (UID: "c2d0b044-0bac-4781-a752-1ec6a575208e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 16:10:37.820978 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:10:37.820958 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2d0b044-0bac-4781-a752-1ec6a575208e-kube-api-access-k4bdd" (OuterVolumeSpecName: "kube-api-access-k4bdd") pod "c2d0b044-0bac-4781-a752-1ec6a575208e" (UID: "c2d0b044-0bac-4781-a752-1ec6a575208e"). InnerVolumeSpecName "kube-api-access-k4bdd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 16:10:37.920032 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:10:37.919990 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k4bdd\" (UniqueName: \"kubernetes.io/projected/c2d0b044-0bac-4781-a752-1ec6a575208e-kube-api-access-k4bdd\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 07 16:10:37.920032 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:10:37.920037 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c2d0b044-0bac-4781-a752-1ec6a575208e-maas-api-tls\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 07 16:10:38.027875 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:10:38.027785 2565 generic.go:356] "Generic (PLEG): container finished" podID="c2d0b044-0bac-4781-a752-1ec6a575208e" containerID="2e2648e0e320749eb2f2fcb848f1bd4b803434e5715cf534f53dc93093ea5e28" exitCode=0 Jul 07 16:10:38.027875 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:10:38.027832 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-cf3310ce-7d5d4f87ff-c7cx9" Jul 07 16:10:38.027875 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:10:38.027854 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-cf3310ce-7d5d4f87ff-c7cx9" event={"ID":"c2d0b044-0bac-4781-a752-1ec6a575208e","Type":"ContainerDied","Data":"2e2648e0e320749eb2f2fcb848f1bd4b803434e5715cf534f53dc93093ea5e28"} Jul 07 16:10:38.028151 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:10:38.027910 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-cf3310ce-7d5d4f87ff-c7cx9" event={"ID":"c2d0b044-0bac-4781-a752-1ec6a575208e","Type":"ContainerDied","Data":"933ab327384b6650be34ea3864acb30eb0e9fe1bb37dbb590e7b773b8b494224"} Jul 07 16:10:38.028151 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:10:38.027938 2565 scope.go:117] "RemoveContainer" containerID="2e2648e0e320749eb2f2fcb848f1bd4b803434e5715cf534f53dc93093ea5e28" Jul 07 16:10:38.037037 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:10:38.037018 2565 scope.go:117] "RemoveContainer" containerID="2e2648e0e320749eb2f2fcb848f1bd4b803434e5715cf534f53dc93093ea5e28" Jul 07 16:10:38.037290 ip-10-0-132-65 kubenswrapper[2565]: E0707 16:10:38.037274 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e2648e0e320749eb2f2fcb848f1bd4b803434e5715cf534f53dc93093ea5e28\": container with ID starting with 2e2648e0e320749eb2f2fcb848f1bd4b803434e5715cf534f53dc93093ea5e28 not found: ID does not exist" containerID="2e2648e0e320749eb2f2fcb848f1bd4b803434e5715cf534f53dc93093ea5e28" Jul 07 16:10:38.037330 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:10:38.037298 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e2648e0e320749eb2f2fcb848f1bd4b803434e5715cf534f53dc93093ea5e28"} err="failed to get container status \"2e2648e0e320749eb2f2fcb848f1bd4b803434e5715cf534f53dc93093ea5e28\": rpc error: code = NotFound desc = could not find container \"2e2648e0e320749eb2f2fcb848f1bd4b803434e5715cf534f53dc93093ea5e28\": container with ID starting with 2e2648e0e320749eb2f2fcb848f1bd4b803434e5715cf534f53dc93093ea5e28 not found: ID does not exist" Jul 07 16:10:38.055223 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:10:38.055196 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-mt-cf3310ce-7d5d4f87ff-c7cx9"] Jul 07 16:10:38.062674 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:10:38.062643 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-mt-cf3310ce-7d5d4f87ff-c7cx9"] Jul 07 16:10:38.187704 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:10:38.187673 2565 patch_prober.go:28] interesting pod/e2e-mt-cf3310ce-openshift-default-94dd57675-njq8z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": read tcp 10.133.0.2:56428->10.133.0.54:15021: read: connection reset by peer" start-of-body= Jul 07 16:10:38.187902 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:10:38.187754 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-cf3310ce-openshift-default-94dd57675-njq8z" podUID="34fd2ca7-0000-4409-ad58-9253a81c5ba4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": read tcp 10.133.0.2:56428->10.133.0.54:15021: read: connection reset by peer" Jul 07 16:10:39.034975 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:10:39.034942 2565 generic.go:356] "Generic (PLEG): container finished" podID="34fd2ca7-0000-4409-ad58-9253a81c5ba4" containerID="1c193ecf4b390590cea76a91b7c920cd4a0b7eb2ba89f1be8a89e1677bd53f91" exitCode=0 Jul 07 16:10:39.035404 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:10:39.035013 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-cf3310ce-openshift-default-94dd57675-njq8z" event={"ID":"34fd2ca7-0000-4409-ad58-9253a81c5ba4","Type":"ContainerDied","Data":"1c193ecf4b390590cea76a91b7c920cd4a0b7eb2ba89f1be8a89e1677bd53f91"} Jul 07 16:10:39.035529 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:10:39.035513 2565 scope.go:117] "RemoveContainer" containerID="1c193ecf4b390590cea76a91b7c920cd4a0b7eb2ba89f1be8a89e1677bd53f91" Jul 07 16:10:39.036531 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:10:39.036487 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 07 16:10:39.036608 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:10:39.036572 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 07 16:10:39.036608 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:10:39.036604 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 07 16:10:39.663103 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:10:39.663066 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c2d0b044-0bac-4781-a752-1ec6a575208e" path="/var/lib/kubelet/pods/c2d0b044-0bac-4781-a752-1ec6a575208e/volumes" Jul 07 16:10:40.042642 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:10:40.042554 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-cf3310ce-openshift-default-94dd57675-njq8z" event={"ID":"34fd2ca7-0000-4409-ad58-9253a81c5ba4","Type":"ContainerStarted","Data":"2bf13e69c01bf809a87fdf4e09718ceea2a1f3829118986136e7aa60f4435606"} Jul 07 16:10:40.281826 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:10:40.281785 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-cf3310ce-openshift-default-94dd57675-njq8z" Jul 07 16:10:40.282209 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:10:40.282179 2565 patch_prober.go:28] interesting pod/e2e-mt-cf3310ce-openshift-default-94dd57675-njq8z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Jul 07 16:10:40.282415 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:10:40.282387 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-cf3310ce-openshift-default-94dd57675-njq8z" podUID="34fd2ca7-0000-4409-ad58-9253a81c5ba4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Jul 07 16:10:41.281955 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:10:41.281903 2565 patch_prober.go:28] interesting pod/e2e-mt-cf3310ce-openshift-default-94dd57675-njq8z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Jul 07 16:10:41.282483 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:10:41.281978 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-cf3310ce-openshift-default-94dd57675-njq8z" podUID="34fd2ca7-0000-4409-ad58-9253a81c5ba4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Jul 07 16:10:42.282274 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:10:42.282236 2565 patch_prober.go:28] interesting pod/e2e-mt-cf3310ce-openshift-default-94dd57675-njq8z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Jul 07 16:10:42.282884 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:10:42.282850 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-cf3310ce-openshift-default-94dd57675-njq8z" podUID="34fd2ca7-0000-4409-ad58-9253a81c5ba4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Jul 07 16:10:43.282060 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:10:43.282022 2565 patch_prober.go:28] interesting pod/e2e-mt-cf3310ce-openshift-default-94dd57675-njq8z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Jul 07 16:10:43.282416 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:10:43.282382 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-cf3310ce-openshift-default-94dd57675-njq8z" podUID="34fd2ca7-0000-4409-ad58-9253a81c5ba4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Jul 07 16:10:44.285051 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:10:44.285017 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-cf3310ce-openshift-default-94dd57675-njq8z" Jul 07 16:10:44.285432 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:10:44.285389 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-cf3310ce-openshift-default-94dd57675-njq8z" Jul 07 16:10:44.286127 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:10:44.286105 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-cf3310ce-openshift-default-94dd57675-njq8z" Jul 07 16:10:44.709717 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:10:44.709683 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:10:44.795902 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:10:44.795873 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:10:45.009086 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:10:45.009001 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-cf3310ce-openshift-default-94dd57675-njq8z"] Jul 07 16:10:45.019601 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:10:45.019573 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:10:45.068219 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:10:45.068178 2565 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/e2e-mt-cf3310ce-openshift-default-94dd57675-njq8z" secret="" err="secret \"e2e-mt-cf3310ce-openshift-default-dockercfg-vv5ht\" not found" Jul 07 16:10:45.263657 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:10:45.263574 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:10:45.380242 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:10:45.380212 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:10:45.474006 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:10:45.473977 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:10:45.610063 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:10:45.610030 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:10:46.072548 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:10:46.072512 2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-cf3310ce-openshift-default-94dd57675-njq8z" podUID="34fd2ca7-0000-4409-ad58-9253a81c5ba4" containerName="istio-proxy" containerID="cri-o://2bf13e69c01bf809a87fdf4e09718ceea2a1f3829118986136e7aa60f4435606" gracePeriod=30 Jul 07 16:10:51.332253 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:10:51.332229 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-cf3310ce-openshift-default-94dd57675-njq8z" Jul 07 16:10:51.450256 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:10:51.450138 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7swfd\" (UniqueName: \"kubernetes.io/projected/34fd2ca7-0000-4409-ad58-9253a81c5ba4-kube-api-access-7swfd\") pod \"34fd2ca7-0000-4409-ad58-9253a81c5ba4\" (UID: \"34fd2ca7-0000-4409-ad58-9253a81c5ba4\") " Jul 07 16:10:51.450256 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:10:51.450195 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34fd2ca7-0000-4409-ad58-9253a81c5ba4-workload-socket\") pod \"34fd2ca7-0000-4409-ad58-9253a81c5ba4\" (UID: \"34fd2ca7-0000-4409-ad58-9253a81c5ba4\") " Jul 07 16:10:51.450256 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:10:51.450238 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34fd2ca7-0000-4409-ad58-9253a81c5ba4-workload-certs\") pod \"34fd2ca7-0000-4409-ad58-9253a81c5ba4\" (UID: \"34fd2ca7-0000-4409-ad58-9253a81c5ba4\") " Jul 07 16:10:51.450595 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:10:51.450270 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34fd2ca7-0000-4409-ad58-9253a81c5ba4-istio-token\") pod \"34fd2ca7-0000-4409-ad58-9253a81c5ba4\" (UID: \"34fd2ca7-0000-4409-ad58-9253a81c5ba4\") " Jul 07 16:10:51.450595 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:10:51.450338 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34fd2ca7-0000-4409-ad58-9253a81c5ba4-istio-data\") pod \"34fd2ca7-0000-4409-ad58-9253a81c5ba4\" (UID: \"34fd2ca7-0000-4409-ad58-9253a81c5ba4\") " Jul 07 16:10:51.450595 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:10:51.450395 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34fd2ca7-0000-4409-ad58-9253a81c5ba4-istiod-ca-cert\") pod \"34fd2ca7-0000-4409-ad58-9253a81c5ba4\" (UID: \"34fd2ca7-0000-4409-ad58-9253a81c5ba4\") " Jul 07 16:10:51.450595 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:10:51.450423 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34fd2ca7-0000-4409-ad58-9253a81c5ba4-istio-podinfo\") pod \"34fd2ca7-0000-4409-ad58-9253a81c5ba4\" (UID: \"34fd2ca7-0000-4409-ad58-9253a81c5ba4\") " Jul 07 16:10:51.450595 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:10:51.450462 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34fd2ca7-0000-4409-ad58-9253a81c5ba4-credential-socket\") pod \"34fd2ca7-0000-4409-ad58-9253a81c5ba4\" (UID: \"34fd2ca7-0000-4409-ad58-9253a81c5ba4\") " Jul 07 16:10:51.450595 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:10:51.450487 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34fd2ca7-0000-4409-ad58-9253a81c5ba4-istio-envoy\") pod \"34fd2ca7-0000-4409-ad58-9253a81c5ba4\" (UID: \"34fd2ca7-0000-4409-ad58-9253a81c5ba4\") " Jul 07 16:10:51.450595 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:10:51.450551 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34fd2ca7-0000-4409-ad58-9253a81c5ba4-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "34fd2ca7-0000-4409-ad58-9253a81c5ba4" (UID: "34fd2ca7-0000-4409-ad58-9253a81c5ba4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 16:10:51.450958 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:10:51.450610 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34fd2ca7-0000-4409-ad58-9253a81c5ba4-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "34fd2ca7-0000-4409-ad58-9253a81c5ba4" (UID: "34fd2ca7-0000-4409-ad58-9253a81c5ba4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 16:10:51.450958 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:10:51.450781 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34fd2ca7-0000-4409-ad58-9253a81c5ba4-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "34fd2ca7-0000-4409-ad58-9253a81c5ba4" (UID: "34fd2ca7-0000-4409-ad58-9253a81c5ba4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 16:10:51.450958 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:10:51.450870 2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34fd2ca7-0000-4409-ad58-9253a81c5ba4-credential-socket\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 07 16:10:51.450958 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:10:51.450889 2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34fd2ca7-0000-4409-ad58-9253a81c5ba4-workload-socket\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 07 16:10:51.450958 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:10:51.450904 2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34fd2ca7-0000-4409-ad58-9253a81c5ba4-workload-certs\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 07 16:10:51.451231 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:10:51.451101 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34fd2ca7-0000-4409-ad58-9253a81c5ba4-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "34fd2ca7-0000-4409-ad58-9253a81c5ba4" (UID: "34fd2ca7-0000-4409-ad58-9253a81c5ba4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 16:10:51.451355 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:10:51.451335 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34fd2ca7-0000-4409-ad58-9253a81c5ba4-istio-data" (OuterVolumeSpecName: "istio-data") pod "34fd2ca7-0000-4409-ad58-9253a81c5ba4" (UID: "34fd2ca7-0000-4409-ad58-9253a81c5ba4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 16:10:51.452454 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:10:51.452428 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34fd2ca7-0000-4409-ad58-9253a81c5ba4-kube-api-access-7swfd" (OuterVolumeSpecName: "kube-api-access-7swfd") pod "34fd2ca7-0000-4409-ad58-9253a81c5ba4" (UID: "34fd2ca7-0000-4409-ad58-9253a81c5ba4"). InnerVolumeSpecName "kube-api-access-7swfd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 16:10:51.452668 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:10:51.452646 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/34fd2ca7-0000-4409-ad58-9253a81c5ba4-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "34fd2ca7-0000-4409-ad58-9253a81c5ba4" (UID: "34fd2ca7-0000-4409-ad58-9253a81c5ba4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 07 16:10:51.452808 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:10:51.452792 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34fd2ca7-0000-4409-ad58-9253a81c5ba4-istio-token" (OuterVolumeSpecName: "istio-token") pod "34fd2ca7-0000-4409-ad58-9253a81c5ba4" (UID: "34fd2ca7-0000-4409-ad58-9253a81c5ba4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 16:10:51.452937 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:10:51.452920 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34fd2ca7-0000-4409-ad58-9253a81c5ba4-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "34fd2ca7-0000-4409-ad58-9253a81c5ba4" (UID: "34fd2ca7-0000-4409-ad58-9253a81c5ba4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 16:10:51.552390 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:10:51.552339 2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34fd2ca7-0000-4409-ad58-9253a81c5ba4-istio-data\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 07 16:10:51.552390 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:10:51.552384 2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34fd2ca7-0000-4409-ad58-9253a81c5ba4-istiod-ca-cert\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 07 16:10:51.552390 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:10:51.552398 2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34fd2ca7-0000-4409-ad58-9253a81c5ba4-istio-podinfo\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 07 16:10:51.552687 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:10:51.552412 2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34fd2ca7-0000-4409-ad58-9253a81c5ba4-istio-envoy\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 07 16:10:51.552687 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:10:51.552423 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7swfd\" (UniqueName: \"kubernetes.io/projected/34fd2ca7-0000-4409-ad58-9253a81c5ba4-kube-api-access-7swfd\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 07 16:10:51.552687 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:10:51.552436 2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34fd2ca7-0000-4409-ad58-9253a81c5ba4-istio-token\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 07 16:10:52.105172 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:10:52.105137 2565 generic.go:356] "Generic (PLEG): container finished" podID="34fd2ca7-0000-4409-ad58-9253a81c5ba4" containerID="2bf13e69c01bf809a87fdf4e09718ceea2a1f3829118986136e7aa60f4435606" exitCode=0 Jul 07 16:10:52.105367 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:10:52.105180 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-cf3310ce-openshift-default-94dd57675-njq8z" event={"ID":"34fd2ca7-0000-4409-ad58-9253a81c5ba4","Type":"ContainerDied","Data":"2bf13e69c01bf809a87fdf4e09718ceea2a1f3829118986136e7aa60f4435606"} Jul 07 16:10:52.105367 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:10:52.105199 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-cf3310ce-openshift-default-94dd57675-njq8z" Jul 07 16:10:52.105367 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:10:52.105233 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-cf3310ce-openshift-default-94dd57675-njq8z" event={"ID":"34fd2ca7-0000-4409-ad58-9253a81c5ba4","Type":"ContainerDied","Data":"a3ecbd3d311209da2edddcc881cb3e920e2996e97a702b038c5e5f45c7ca7839"} Jul 07 16:10:52.105367 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:10:52.105261 2565 scope.go:117] "RemoveContainer" containerID="2bf13e69c01bf809a87fdf4e09718ceea2a1f3829118986136e7aa60f4435606" Jul 07 16:10:52.114182 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:10:52.114155 2565 scope.go:117] "RemoveContainer" containerID="1c193ecf4b390590cea76a91b7c920cd4a0b7eb2ba89f1be8a89e1677bd53f91" Jul 07 16:10:52.122456 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:10:52.122428 2565 scope.go:117] "RemoveContainer" containerID="2bf13e69c01bf809a87fdf4e09718ceea2a1f3829118986136e7aa60f4435606" Jul 07 16:10:52.122750 ip-10-0-132-65 kubenswrapper[2565]: E0707 16:10:52.122722 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2bf13e69c01bf809a87fdf4e09718ceea2a1f3829118986136e7aa60f4435606\": container with ID starting with 2bf13e69c01bf809a87fdf4e09718ceea2a1f3829118986136e7aa60f4435606 not found: ID does not exist" containerID="2bf13e69c01bf809a87fdf4e09718ceea2a1f3829118986136e7aa60f4435606" Jul 07 16:10:52.122821 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:10:52.122752 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2bf13e69c01bf809a87fdf4e09718ceea2a1f3829118986136e7aa60f4435606"} err="failed to get container status \"2bf13e69c01bf809a87fdf4e09718ceea2a1f3829118986136e7aa60f4435606\": rpc error: code = NotFound desc = could not find container \"2bf13e69c01bf809a87fdf4e09718ceea2a1f3829118986136e7aa60f4435606\": container with ID starting with 2bf13e69c01bf809a87fdf4e09718ceea2a1f3829118986136e7aa60f4435606 not found: ID does not exist" Jul 07 16:10:52.122821 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:10:52.122770 2565 scope.go:117] "RemoveContainer" containerID="1c193ecf4b390590cea76a91b7c920cd4a0b7eb2ba89f1be8a89e1677bd53f91" Jul 07 16:10:52.123013 ip-10-0-132-65 kubenswrapper[2565]: E0707 16:10:52.122990 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c193ecf4b390590cea76a91b7c920cd4a0b7eb2ba89f1be8a89e1677bd53f91\": container with ID starting with 1c193ecf4b390590cea76a91b7c920cd4a0b7eb2ba89f1be8a89e1677bd53f91 not found: ID does not exist" containerID="1c193ecf4b390590cea76a91b7c920cd4a0b7eb2ba89f1be8a89e1677bd53f91" Jul 07 16:10:52.123102 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:10:52.123019 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c193ecf4b390590cea76a91b7c920cd4a0b7eb2ba89f1be8a89e1677bd53f91"} err="failed to get container status \"1c193ecf4b390590cea76a91b7c920cd4a0b7eb2ba89f1be8a89e1677bd53f91\": rpc error: code = NotFound desc = could not find container \"1c193ecf4b390590cea76a91b7c920cd4a0b7eb2ba89f1be8a89e1677bd53f91\": container with ID starting with 1c193ecf4b390590cea76a91b7c920cd4a0b7eb2ba89f1be8a89e1677bd53f91 not found: ID does not exist" Jul 07 16:10:52.130574 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:10:52.130551 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-cf3310ce-openshift-default-94dd57675-njq8z"] Jul 07 16:10:52.138315 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:10:52.138293 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-cf3310ce-openshift-default-94dd57675-njq8z"] Jul 07 16:10:52.586905 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:10:52.586875 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:10:52.688726 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:10:52.688695 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:10:52.847787 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:10:52.847700 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:10:52.921235 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:10:52.921206 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:10:53.009867 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:10:53.009832 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:10:53.110259 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:10:53.110164 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:10:53.269087 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:10:53.269058 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:10:53.661203 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:10:53.661168 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="34fd2ca7-0000-4409-ad58-9253a81c5ba4" path="/var/lib/kubelet/pods/34fd2ca7-0000-4409-ad58-9253a81c5ba4/volumes" Jul 07 16:11:00.733363 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:11:00.733329 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:11:00.815289 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:11:00.815252 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:11:01.634563 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:11:01.634471 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-s9kl8_62da0165-d32c-4bba-9844-a49154d765e5/console-operator/1.log" Jul 07 16:11:01.636225 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:11:01.636199 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-s9kl8_62da0165-d32c-4bba-9844-a49154d765e5/console-operator/1.log" Jul 07 16:11:01.639301 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:11:01.639274 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-942xh_31ff16f0-6054-4eba-942a-b07a5a7afac7/ovn-acl-logging/0.log" Jul 07 16:11:01.640756 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:11:01.640737 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-942xh_31ff16f0-6054-4eba-942a-b07a5a7afac7/ovn-acl-logging/0.log" Jul 07 16:11:07.976371 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:11:07.976334 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:11:08.344382 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:11:08.344338 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-de210ed2-openshift-default-7b667c4c5f-k6cgm"] Jul 07 16:11:08.344739 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:11:08.344724 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="34fd2ca7-0000-4409-ad58-9253a81c5ba4" containerName="istio-proxy" Jul 07 16:11:08.344739 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:11:08.344738 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="34fd2ca7-0000-4409-ad58-9253a81c5ba4" containerName="istio-proxy" Jul 07 16:11:08.344887 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:11:08.344750 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="34fd2ca7-0000-4409-ad58-9253a81c5ba4" containerName="istio-proxy" Jul 07 16:11:08.344887 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:11:08.344755 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="34fd2ca7-0000-4409-ad58-9253a81c5ba4" containerName="istio-proxy" Jul 07 16:11:08.344887 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:11:08.344770 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c2d0b044-0bac-4781-a752-1ec6a575208e" containerName="maas-api" Jul 07 16:11:08.344887 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:11:08.344776 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="c2d0b044-0bac-4781-a752-1ec6a575208e" containerName="maas-api" Jul 07 16:11:08.344887 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:11:08.344835 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="c2d0b044-0bac-4781-a752-1ec6a575208e" containerName="maas-api" Jul 07 16:11:08.344887 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:11:08.344845 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="34fd2ca7-0000-4409-ad58-9253a81c5ba4" containerName="istio-proxy" Jul 07 16:11:08.346850 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:11:08.346828 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-de210ed2-openshift-default-7b667c4c5f-k6cgm" Jul 07 16:11:08.352371 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:11:08.352348 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-de210ed2-openshift-default-dockercfg-tjtnq\"" Jul 07 16:11:08.356781 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:11:08.356761 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:11:08.372217 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:11:08.372181 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-de210ed2-openshift-default-7b667c4c5f-k6cgm"] Jul 07 16:11:08.508784 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:11:08.508746 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40a5a7ce-85bb-43f3-b33e-9af689d3e95c-istiod-ca-cert\") pod \"e2e-mt-de210ed2-openshift-default-7b667c4c5f-k6cgm\" (UID: \"40a5a7ce-85bb-43f3-b33e-9af689d3e95c\") " pod="openshift-ingress/e2e-mt-de210ed2-openshift-default-7b667c4c5f-k6cgm" Jul 07 16:11:08.508988 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:11:08.508796 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40a5a7ce-85bb-43f3-b33e-9af689d3e95c-istio-data\") pod \"e2e-mt-de210ed2-openshift-default-7b667c4c5f-k6cgm\" (UID: \"40a5a7ce-85bb-43f3-b33e-9af689d3e95c\") " pod="openshift-ingress/e2e-mt-de210ed2-openshift-default-7b667c4c5f-k6cgm" Jul 07 16:11:08.508988 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:11:08.508824 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40a5a7ce-85bb-43f3-b33e-9af689d3e95c-workload-socket\") pod \"e2e-mt-de210ed2-openshift-default-7b667c4c5f-k6cgm\" (UID: \"40a5a7ce-85bb-43f3-b33e-9af689d3e95c\") " pod="openshift-ingress/e2e-mt-de210ed2-openshift-default-7b667c4c5f-k6cgm" Jul 07 16:11:08.508988 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:11:08.508896 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xsm5q\" (UniqueName: \"kubernetes.io/projected/40a5a7ce-85bb-43f3-b33e-9af689d3e95c-kube-api-access-xsm5q\") pod \"e2e-mt-de210ed2-openshift-default-7b667c4c5f-k6cgm\" (UID: \"40a5a7ce-85bb-43f3-b33e-9af689d3e95c\") " pod="openshift-ingress/e2e-mt-de210ed2-openshift-default-7b667c4c5f-k6cgm" Jul 07 16:11:08.508988 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:11:08.508945 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40a5a7ce-85bb-43f3-b33e-9af689d3e95c-istio-envoy\") pod \"e2e-mt-de210ed2-openshift-default-7b667c4c5f-k6cgm\" (UID: \"40a5a7ce-85bb-43f3-b33e-9af689d3e95c\") " pod="openshift-ingress/e2e-mt-de210ed2-openshift-default-7b667c4c5f-k6cgm" Jul 07 16:11:08.508988 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:11:08.508982 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40a5a7ce-85bb-43f3-b33e-9af689d3e95c-workload-certs\") pod \"e2e-mt-de210ed2-openshift-default-7b667c4c5f-k6cgm\" (UID: \"40a5a7ce-85bb-43f3-b33e-9af689d3e95c\") " pod="openshift-ingress/e2e-mt-de210ed2-openshift-default-7b667c4c5f-k6cgm" Jul 07 16:11:08.509247 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:11:08.508999 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40a5a7ce-85bb-43f3-b33e-9af689d3e95c-istio-podinfo\") pod \"e2e-mt-de210ed2-openshift-default-7b667c4c5f-k6cgm\" (UID: \"40a5a7ce-85bb-43f3-b33e-9af689d3e95c\") " pod="openshift-ingress/e2e-mt-de210ed2-openshift-default-7b667c4c5f-k6cgm" Jul 07 16:11:08.509247 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:11:08.509038 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40a5a7ce-85bb-43f3-b33e-9af689d3e95c-istio-token\") pod \"e2e-mt-de210ed2-openshift-default-7b667c4c5f-k6cgm\" (UID: \"40a5a7ce-85bb-43f3-b33e-9af689d3e95c\") " pod="openshift-ingress/e2e-mt-de210ed2-openshift-default-7b667c4c5f-k6cgm" Jul 07 16:11:08.509247 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:11:08.509059 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40a5a7ce-85bb-43f3-b33e-9af689d3e95c-credential-socket\") pod \"e2e-mt-de210ed2-openshift-default-7b667c4c5f-k6cgm\" (UID: \"40a5a7ce-85bb-43f3-b33e-9af689d3e95c\") " pod="openshift-ingress/e2e-mt-de210ed2-openshift-default-7b667c4c5f-k6cgm" Jul 07 16:11:08.610665 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:11:08.610568 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40a5a7ce-85bb-43f3-b33e-9af689d3e95c-workload-socket\") pod \"e2e-mt-de210ed2-openshift-default-7b667c4c5f-k6cgm\" (UID: \"40a5a7ce-85bb-43f3-b33e-9af689d3e95c\") " pod="openshift-ingress/e2e-mt-de210ed2-openshift-default-7b667c4c5f-k6cgm" Jul 07 16:11:08.610665 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:11:08.610610 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xsm5q\" (UniqueName: \"kubernetes.io/projected/40a5a7ce-85bb-43f3-b33e-9af689d3e95c-kube-api-access-xsm5q\") pod \"e2e-mt-de210ed2-openshift-default-7b667c4c5f-k6cgm\" (UID: \"40a5a7ce-85bb-43f3-b33e-9af689d3e95c\") " pod="openshift-ingress/e2e-mt-de210ed2-openshift-default-7b667c4c5f-k6cgm" Jul 07 16:11:08.610665 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:11:08.610641 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40a5a7ce-85bb-43f3-b33e-9af689d3e95c-istio-envoy\") pod \"e2e-mt-de210ed2-openshift-default-7b667c4c5f-k6cgm\" (UID: \"40a5a7ce-85bb-43f3-b33e-9af689d3e95c\") " pod="openshift-ingress/e2e-mt-de210ed2-openshift-default-7b667c4c5f-k6cgm" Jul 07 16:11:08.610939 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:11:08.610676 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40a5a7ce-85bb-43f3-b33e-9af689d3e95c-workload-certs\") pod \"e2e-mt-de210ed2-openshift-default-7b667c4c5f-k6cgm\" (UID: \"40a5a7ce-85bb-43f3-b33e-9af689d3e95c\") " pod="openshift-ingress/e2e-mt-de210ed2-openshift-default-7b667c4c5f-k6cgm" Jul 07 16:11:08.610939 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:11:08.610692 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40a5a7ce-85bb-43f3-b33e-9af689d3e95c-istio-podinfo\") pod \"e2e-mt-de210ed2-openshift-default-7b667c4c5f-k6cgm\" (UID: \"40a5a7ce-85bb-43f3-b33e-9af689d3e95c\") " pod="openshift-ingress/e2e-mt-de210ed2-openshift-default-7b667c4c5f-k6cgm" Jul 07 16:11:08.610939 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:11:08.610848 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40a5a7ce-85bb-43f3-b33e-9af689d3e95c-istio-token\") pod \"e2e-mt-de210ed2-openshift-default-7b667c4c5f-k6cgm\" (UID: \"40a5a7ce-85bb-43f3-b33e-9af689d3e95c\") " pod="openshift-ingress/e2e-mt-de210ed2-openshift-default-7b667c4c5f-k6cgm" Jul 07 16:11:08.610939 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:11:08.610881 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40a5a7ce-85bb-43f3-b33e-9af689d3e95c-credential-socket\") pod \"e2e-mt-de210ed2-openshift-default-7b667c4c5f-k6cgm\" (UID: \"40a5a7ce-85bb-43f3-b33e-9af689d3e95c\") " pod="openshift-ingress/e2e-mt-de210ed2-openshift-default-7b667c4c5f-k6cgm" Jul 07 16:11:08.611130 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:11:08.610967 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40a5a7ce-85bb-43f3-b33e-9af689d3e95c-istiod-ca-cert\") pod \"e2e-mt-de210ed2-openshift-default-7b667c4c5f-k6cgm\" (UID: \"40a5a7ce-85bb-43f3-b33e-9af689d3e95c\") " pod="openshift-ingress/e2e-mt-de210ed2-openshift-default-7b667c4c5f-k6cgm" Jul 07 16:11:08.611130 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:11:08.611017 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40a5a7ce-85bb-43f3-b33e-9af689d3e95c-istio-data\") pod \"e2e-mt-de210ed2-openshift-default-7b667c4c5f-k6cgm\" (UID: \"40a5a7ce-85bb-43f3-b33e-9af689d3e95c\") " pod="openshift-ingress/e2e-mt-de210ed2-openshift-default-7b667c4c5f-k6cgm" Jul 07 16:11:08.611130 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:11:08.611107 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40a5a7ce-85bb-43f3-b33e-9af689d3e95c-workload-certs\") pod \"e2e-mt-de210ed2-openshift-default-7b667c4c5f-k6cgm\" (UID: \"40a5a7ce-85bb-43f3-b33e-9af689d3e95c\") " pod="openshift-ingress/e2e-mt-de210ed2-openshift-default-7b667c4c5f-k6cgm" Jul 07 16:11:08.611519 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:11:08.611359 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40a5a7ce-85bb-43f3-b33e-9af689d3e95c-credential-socket\") pod \"e2e-mt-de210ed2-openshift-default-7b667c4c5f-k6cgm\" (UID: \"40a5a7ce-85bb-43f3-b33e-9af689d3e95c\") " pod="openshift-ingress/e2e-mt-de210ed2-openshift-default-7b667c4c5f-k6cgm" Jul 07 16:11:08.611519 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:11:08.611444 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40a5a7ce-85bb-43f3-b33e-9af689d3e95c-istio-data\") pod \"e2e-mt-de210ed2-openshift-default-7b667c4c5f-k6cgm\" (UID: \"40a5a7ce-85bb-43f3-b33e-9af689d3e95c\") " pod="openshift-ingress/e2e-mt-de210ed2-openshift-default-7b667c4c5f-k6cgm" Jul 07 16:11:08.611690 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:11:08.611640 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40a5a7ce-85bb-43f3-b33e-9af689d3e95c-workload-socket\") pod \"e2e-mt-de210ed2-openshift-default-7b667c4c5f-k6cgm\" (UID: \"40a5a7ce-85bb-43f3-b33e-9af689d3e95c\") " pod="openshift-ingress/e2e-mt-de210ed2-openshift-default-7b667c4c5f-k6cgm" Jul 07 16:11:08.611892 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:11:08.611870 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40a5a7ce-85bb-43f3-b33e-9af689d3e95c-istiod-ca-cert\") pod \"e2e-mt-de210ed2-openshift-default-7b667c4c5f-k6cgm\" (UID: \"40a5a7ce-85bb-43f3-b33e-9af689d3e95c\") " pod="openshift-ingress/e2e-mt-de210ed2-openshift-default-7b667c4c5f-k6cgm" Jul 07 16:11:08.613009 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:11:08.612990 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40a5a7ce-85bb-43f3-b33e-9af689d3e95c-istio-envoy\") pod \"e2e-mt-de210ed2-openshift-default-7b667c4c5f-k6cgm\" (UID: \"40a5a7ce-85bb-43f3-b33e-9af689d3e95c\") " pod="openshift-ingress/e2e-mt-de210ed2-openshift-default-7b667c4c5f-k6cgm" Jul 07 16:11:08.613449 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:11:08.613431 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40a5a7ce-85bb-43f3-b33e-9af689d3e95c-istio-podinfo\") pod \"e2e-mt-de210ed2-openshift-default-7b667c4c5f-k6cgm\" (UID: \"40a5a7ce-85bb-43f3-b33e-9af689d3e95c\") " pod="openshift-ingress/e2e-mt-de210ed2-openshift-default-7b667c4c5f-k6cgm" Jul 07 16:11:08.622211 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:11:08.622165 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40a5a7ce-85bb-43f3-b33e-9af689d3e95c-istio-token\") pod \"e2e-mt-de210ed2-openshift-default-7b667c4c5f-k6cgm\" (UID: \"40a5a7ce-85bb-43f3-b33e-9af689d3e95c\") " pod="openshift-ingress/e2e-mt-de210ed2-openshift-default-7b667c4c5f-k6cgm" Jul 07 16:11:08.623281 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:11:08.623259 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xsm5q\" (UniqueName: \"kubernetes.io/projected/40a5a7ce-85bb-43f3-b33e-9af689d3e95c-kube-api-access-xsm5q\") pod \"e2e-mt-de210ed2-openshift-default-7b667c4c5f-k6cgm\" (UID: \"40a5a7ce-85bb-43f3-b33e-9af689d3e95c\") " pod="openshift-ingress/e2e-mt-de210ed2-openshift-default-7b667c4c5f-k6cgm" Jul 07 16:11:08.658363 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:11:08.658318 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-de210ed2-openshift-default-7b667c4c5f-k6cgm" Jul 07 16:11:08.816468 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:11:08.816442 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-de210ed2-openshift-default-7b667c4c5f-k6cgm"] Jul 07 16:11:08.817866 ip-10-0-132-65 kubenswrapper[2565]: W0707 16:11:08.817840 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40a5a7ce_85bb_43f3_b33e_9af689d3e95c.slice/crio-aab690cd0df8f5ae912041078c8d123d23cba4b2b8d282b3ce3089bcbb1a9f15 WatchSource:0}: Error finding container aab690cd0df8f5ae912041078c8d123d23cba4b2b8d282b3ce3089bcbb1a9f15: Status 404 returned error can't find the container with id aab690cd0df8f5ae912041078c8d123d23cba4b2b8d282b3ce3089bcbb1a9f15 Jul 07 16:11:08.819892 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:11:08.819862 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 07 16:11:08.819966 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:11:08.819923 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 07 16:11:08.819966 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:11:08.819954 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 07 16:11:08.976539 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:11:08.976421 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:11:09.191373 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:11:09.191284 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-de210ed2-openshift-default-7b667c4c5f-k6cgm" event={"ID":"40a5a7ce-85bb-43f3-b33e-9af689d3e95c","Type":"ContainerStarted","Data":"1ebcd6eb0e8c4f77b9c7a0b479d6507c204c9e7c3ed2564c09be9c86d45c346b"} Jul 07 16:11:09.191373 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:11:09.191343 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-de210ed2-openshift-default-7b667c4c5f-k6cgm" event={"ID":"40a5a7ce-85bb-43f3-b33e-9af689d3e95c","Type":"ContainerStarted","Data":"aab690cd0df8f5ae912041078c8d123d23cba4b2b8d282b3ce3089bcbb1a9f15"} Jul 07 16:11:09.221511 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:11:09.221414 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-de210ed2-openshift-default-7b667c4c5f-k6cgm" podStartSLOduration=1.221392903 podStartE2EDuration="1.221392903s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:11:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:11:09.219676073 +0000 UTC m=+1808.176253566" watchObservedRunningTime="2026-07-07 16:11:09.221392903 +0000 UTC m=+1808.177970374" Jul 07 16:11:09.666571 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:11:09.666539 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-de210ed2-openshift-default-7b667c4c5f-k6cgm" Jul 07 16:11:09.667271 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:11:09.667253 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-de210ed2-openshift-default-7b667c4c5f-k6cgm" Jul 07 16:11:10.196616 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:11:10.196585 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-de210ed2-openshift-default-7b667c4c5f-k6cgm" Jul 07 16:11:10.197569 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:11:10.197551 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-de210ed2-openshift-default-7b667c4c5f-k6cgm" Jul 07 16:11:11.193601 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:11:11.193558 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:11:11.296705 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:11:11.296670 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:11:11.297805 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:11:11.297785 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-mt-de210ed2-79fd54f7d8-b8nxq"] Jul 07 16:11:11.298216 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:11:11.298204 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="34fd2ca7-0000-4409-ad58-9253a81c5ba4" containerName="istio-proxy" Jul 07 16:11:11.300560 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:11:11.300545 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-de210ed2-79fd54f7d8-b8nxq" Jul 07 16:11:11.306653 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:11:11.306631 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-mt-de210ed2-serving-cert\"" Jul 07 16:11:11.307194 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:11:11.306970 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-dockercfg-9bqq6\"" Jul 07 16:11:11.317213 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:11:11.317193 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-mt-de210ed2-79fd54f7d8-b8nxq"] Jul 07 16:11:11.385023 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:11:11.384981 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:11:11.439831 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:11:11.439800 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvn9b\" (UniqueName: \"kubernetes.io/projected/1aefdb27-068e-4a46-9407-9baa5bb4182b-kube-api-access-gvn9b\") pod \"maas-api-e2e-mt-de210ed2-79fd54f7d8-b8nxq\" (UID: \"1aefdb27-068e-4a46-9407-9baa5bb4182b\") " pod="opendatahub/maas-api-e2e-mt-de210ed2-79fd54f7d8-b8nxq" Jul 07 16:11:11.440005 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:11:11.439842 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1aefdb27-068e-4a46-9407-9baa5bb4182b-openshift-service-ca\") pod \"maas-api-e2e-mt-de210ed2-79fd54f7d8-b8nxq\" (UID: \"1aefdb27-068e-4a46-9407-9baa5bb4182b\") " pod="opendatahub/maas-api-e2e-mt-de210ed2-79fd54f7d8-b8nxq" Jul 07 16:11:11.440005 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:11:11.439868 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1aefdb27-068e-4a46-9407-9baa5bb4182b-maas-api-tls\") pod \"maas-api-e2e-mt-de210ed2-79fd54f7d8-b8nxq\" (UID: \"1aefdb27-068e-4a46-9407-9baa5bb4182b\") " pod="opendatahub/maas-api-e2e-mt-de210ed2-79fd54f7d8-b8nxq" Jul 07 16:11:11.540995 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:11:11.540892 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvn9b\" (UniqueName: \"kubernetes.io/projected/1aefdb27-068e-4a46-9407-9baa5bb4182b-kube-api-access-gvn9b\") pod \"maas-api-e2e-mt-de210ed2-79fd54f7d8-b8nxq\" (UID: \"1aefdb27-068e-4a46-9407-9baa5bb4182b\") " pod="opendatahub/maas-api-e2e-mt-de210ed2-79fd54f7d8-b8nxq" Jul 07 16:11:11.540995 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:11:11.540947 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1aefdb27-068e-4a46-9407-9baa5bb4182b-openshift-service-ca\") pod \"maas-api-e2e-mt-de210ed2-79fd54f7d8-b8nxq\" (UID: \"1aefdb27-068e-4a46-9407-9baa5bb4182b\") " pod="opendatahub/maas-api-e2e-mt-de210ed2-79fd54f7d8-b8nxq" Jul 07 16:11:11.540995 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:11:11.540985 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1aefdb27-068e-4a46-9407-9baa5bb4182b-maas-api-tls\") pod \"maas-api-e2e-mt-de210ed2-79fd54f7d8-b8nxq\" (UID: \"1aefdb27-068e-4a46-9407-9baa5bb4182b\") " pod="opendatahub/maas-api-e2e-mt-de210ed2-79fd54f7d8-b8nxq" Jul 07 16:11:11.541312 ip-10-0-132-65 kubenswrapper[2565]: E0707 16:11:11.541181 2565 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-mt-de210ed2-serving-cert: secret "maas-api-e2e-mt-de210ed2-serving-cert" not found Jul 07 16:11:11.541312 ip-10-0-132-65 kubenswrapper[2565]: E0707 16:11:11.541253 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1aefdb27-068e-4a46-9407-9baa5bb4182b-maas-api-tls podName:1aefdb27-068e-4a46-9407-9baa5bb4182b nodeName:}" failed. No retries permitted until 2026-07-07 16:11:12.041230656 +0000 UTC m=+1810.997808105 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/1aefdb27-068e-4a46-9407-9baa5bb4182b-maas-api-tls") pod "maas-api-e2e-mt-de210ed2-79fd54f7d8-b8nxq" (UID: "1aefdb27-068e-4a46-9407-9baa5bb4182b") : secret "maas-api-e2e-mt-de210ed2-serving-cert" not found Jul 07 16:11:11.541649 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:11:11.541628 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1aefdb27-068e-4a46-9407-9baa5bb4182b-openshift-service-ca\") pod \"maas-api-e2e-mt-de210ed2-79fd54f7d8-b8nxq\" (UID: \"1aefdb27-068e-4a46-9407-9baa5bb4182b\") " pod="opendatahub/maas-api-e2e-mt-de210ed2-79fd54f7d8-b8nxq" Jul 07 16:11:11.555389 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:11:11.555361 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvn9b\" (UniqueName: \"kubernetes.io/projected/1aefdb27-068e-4a46-9407-9baa5bb4182b-kube-api-access-gvn9b\") pod \"maas-api-e2e-mt-de210ed2-79fd54f7d8-b8nxq\" (UID: \"1aefdb27-068e-4a46-9407-9baa5bb4182b\") " pod="opendatahub/maas-api-e2e-mt-de210ed2-79fd54f7d8-b8nxq" Jul 07 16:11:11.709174 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:11:11.709141 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:11:12.046298 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:11:12.046260 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1aefdb27-068e-4a46-9407-9baa5bb4182b-maas-api-tls\") pod \"maas-api-e2e-mt-de210ed2-79fd54f7d8-b8nxq\" (UID: \"1aefdb27-068e-4a46-9407-9baa5bb4182b\") " pod="opendatahub/maas-api-e2e-mt-de210ed2-79fd54f7d8-b8nxq" Jul 07 16:11:12.048837 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:11:12.048817 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1aefdb27-068e-4a46-9407-9baa5bb4182b-maas-api-tls\") pod \"maas-api-e2e-mt-de210ed2-79fd54f7d8-b8nxq\" (UID: \"1aefdb27-068e-4a46-9407-9baa5bb4182b\") " pod="opendatahub/maas-api-e2e-mt-de210ed2-79fd54f7d8-b8nxq" Jul 07 16:11:12.103740 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:11:12.103710 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:11:12.195463 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:11:12.195429 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:11:12.211162 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:11:12.211134 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-de210ed2-79fd54f7d8-b8nxq" Jul 07 16:11:12.572006 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:11:12.571970 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-mt-de210ed2-79fd54f7d8-b8nxq"] Jul 07 16:11:12.575298 ip-10-0-132-65 kubenswrapper[2565]: W0707 16:11:12.575268 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1aefdb27_068e_4a46_9407_9baa5bb4182b.slice/crio-84c54e38504acf7fe91518858207d3664262f66c1165dc2fa2ff61a1abfe30a9 WatchSource:0}: Error finding container 84c54e38504acf7fe91518858207d3664262f66c1165dc2fa2ff61a1abfe30a9: Status 404 returned error can't find the container with id 84c54e38504acf7fe91518858207d3664262f66c1165dc2fa2ff61a1abfe30a9 Jul 07 16:11:13.212406 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:11:13.212368 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-de210ed2-79fd54f7d8-b8nxq" event={"ID":"1aefdb27-068e-4a46-9407-9baa5bb4182b","Type":"ContainerStarted","Data":"210b0c14b391e4adde3ea46661334ba965a5983966aa966a9d92e77a8e04a787"} Jul 07 16:11:13.212600 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:11:13.212427 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-de210ed2-79fd54f7d8-b8nxq" event={"ID":"1aefdb27-068e-4a46-9407-9baa5bb4182b","Type":"ContainerStarted","Data":"84c54e38504acf7fe91518858207d3664262f66c1165dc2fa2ff61a1abfe30a9"} Jul 07 16:11:13.212600 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:11:13.212481 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-mt-de210ed2-79fd54f7d8-b8nxq" Jul 07 16:11:13.236548 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:11:13.236480 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-mt-de210ed2-79fd54f7d8-b8nxq" podStartSLOduration=1.783478169 podStartE2EDuration="2.236461939s" totalImagesPullingTime="452.98377ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:11:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:11:13.234777461 +0000 UTC m=+1812.191354928" watchObservedRunningTime="2026-07-07 16:11:13.236461939 +0000 UTC m=+1812.193039410" Jul 07 16:11:15.727607 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:11:15.727557 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:11:19.228762 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:11:19.228735 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-mt-de210ed2-79fd54f7d8-b8nxq" Jul 07 16:11:24.273971 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:11:24.273862 2565 generic.go:356] "Generic (PLEG): container finished" podID="40a5a7ce-85bb-43f3-b33e-9af689d3e95c" containerID="1ebcd6eb0e8c4f77b9c7a0b479d6507c204c9e7c3ed2564c09be9c86d45c346b" exitCode=0 Jul 07 16:11:24.273971 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:11:24.273930 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-de210ed2-openshift-default-7b667c4c5f-k6cgm" event={"ID":"40a5a7ce-85bb-43f3-b33e-9af689d3e95c","Type":"ContainerDied","Data":"1ebcd6eb0e8c4f77b9c7a0b479d6507c204c9e7c3ed2564c09be9c86d45c346b"} Jul 07 16:11:24.274403 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:11:24.274324 2565 scope.go:117] "RemoveContainer" containerID="1ebcd6eb0e8c4f77b9c7a0b479d6507c204c9e7c3ed2564c09be9c86d45c346b" Jul 07 16:11:24.275444 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:11:24.275410 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 07 16:11:24.275535 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:11:24.275519 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 07 16:11:24.275592 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:11:24.275563 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 07 16:11:25.196848 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:11:25.196813 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-de210ed2-openshift-default-7b667c4c5f-k6cgm" Jul 07 16:11:25.284028 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:11:25.283987 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-de210ed2-openshift-default-7b667c4c5f-k6cgm" event={"ID":"40a5a7ce-85bb-43f3-b33e-9af689d3e95c","Type":"ContainerStarted","Data":"b0ee929e8a63b5dc5be950f3aaa06bda27924763ead674ec32e6272df0cc4567"} Jul 07 16:11:25.660059 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:11:25.660017 2565 patch_prober.go:28] interesting pod/e2e-mt-de210ed2-openshift-default-7b667c4c5f-k6cgm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Jul 07 16:11:25.660261 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:11:25.660082 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-de210ed2-openshift-default-7b667c4c5f-k6cgm" podUID="40a5a7ce-85bb-43f3-b33e-9af689d3e95c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Jul 07 16:11:25.663282 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:11:25.663249 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-de210ed2-openshift-default-7b667c4c5f-k6cgm" Jul 07 16:11:26.659817 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:11:26.659775 2565 patch_prober.go:28] interesting pod/e2e-mt-de210ed2-openshift-default-7b667c4c5f-k6cgm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Jul 07 16:11:26.660438 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:11:26.660402 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-de210ed2-openshift-default-7b667c4c5f-k6cgm" podUID="40a5a7ce-85bb-43f3-b33e-9af689d3e95c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Jul 07 16:11:27.658751 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:11:27.658713 2565 patch_prober.go:28] interesting pod/e2e-mt-de210ed2-openshift-default-7b667c4c5f-k6cgm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Jul 07 16:11:27.658930 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:11:27.658779 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-de210ed2-openshift-default-7b667c4c5f-k6cgm" podUID="40a5a7ce-85bb-43f3-b33e-9af689d3e95c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Jul 07 16:11:28.659079 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:11:28.659031 2565 patch_prober.go:28] interesting pod/e2e-mt-de210ed2-openshift-default-7b667c4c5f-k6cgm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Jul 07 16:11:28.659621 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:11:28.659101 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-de210ed2-openshift-default-7b667c4c5f-k6cgm" podUID="40a5a7ce-85bb-43f3-b33e-9af689d3e95c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Jul 07 16:11:29.400556 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:11:29.400518 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-de210ed2/e2e-lifecycle-model-de210ed2-kserve-b94d584f9-vfl5v"] Jul 07 16:11:29.403727 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:11:29.403698 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-de210ed2/e2e-lifecycle-model-de210ed2-kserve-b94d584f9-vfl5v" Jul 07 16:11:29.407178 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:11:29.407153 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-de210ed2\"/\"kube-root-ca.crt\"" Jul 07 16:11:29.407327 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:11:29.407311 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-de210ed2\"/\"default-dockercfg-8jthg\"" Jul 07 16:11:29.408392 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:11:29.408348 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-de210ed2\"/\"openshift-service-ca.crt\"" Jul 07 16:11:29.408641 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:11:29.408612 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-de210ed2\"/\"e2e-lifecycle-model-de210ed2-kserve-self-signed-certs\"" Jul 07 16:11:29.422816 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:11:29.422790 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-de210ed2/e2e-lifecycle-model-de210ed2-kserve-b94d584f9-vfl5v"] Jul 07 16:11:29.437928 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:11:29.437878 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:11:29.511833 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:11:29.511798 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d48c310f-8651-42ca-8d78-7344c29e26dd-tls-certs\") pod \"e2e-lifecycle-model-de210ed2-kserve-b94d584f9-vfl5v\" (UID: \"d48c310f-8651-42ca-8d78-7344c29e26dd\") " pod="ai-tenant-e2e-mt-de210ed2/e2e-lifecycle-model-de210ed2-kserve-b94d584f9-vfl5v" Jul 07 16:11:29.511833 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:11:29.511834 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d48c310f-8651-42ca-8d78-7344c29e26dd-home\") pod \"e2e-lifecycle-model-de210ed2-kserve-b94d584f9-vfl5v\" (UID: \"d48c310f-8651-42ca-8d78-7344c29e26dd\") " pod="ai-tenant-e2e-mt-de210ed2/e2e-lifecycle-model-de210ed2-kserve-b94d584f9-vfl5v" Jul 07 16:11:29.512085 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:11:29.511881 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d48c310f-8651-42ca-8d78-7344c29e26dd-model-cache\") pod \"e2e-lifecycle-model-de210ed2-kserve-b94d584f9-vfl5v\" (UID: \"d48c310f-8651-42ca-8d78-7344c29e26dd\") " pod="ai-tenant-e2e-mt-de210ed2/e2e-lifecycle-model-de210ed2-kserve-b94d584f9-vfl5v" Jul 07 16:11:29.512085 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:11:29.511925 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d48c310f-8651-42ca-8d78-7344c29e26dd-tmp-dir\") pod \"e2e-lifecycle-model-de210ed2-kserve-b94d584f9-vfl5v\" (UID: \"d48c310f-8651-42ca-8d78-7344c29e26dd\") " pod="ai-tenant-e2e-mt-de210ed2/e2e-lifecycle-model-de210ed2-kserve-b94d584f9-vfl5v" Jul 07 16:11:29.512085 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:11:29.511962 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d48c310f-8651-42ca-8d78-7344c29e26dd-dshm\") pod \"e2e-lifecycle-model-de210ed2-kserve-b94d584f9-vfl5v\" (UID: \"d48c310f-8651-42ca-8d78-7344c29e26dd\") " pod="ai-tenant-e2e-mt-de210ed2/e2e-lifecycle-model-de210ed2-kserve-b94d584f9-vfl5v" Jul 07 16:11:29.512085 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:11:29.511986 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g22vh\" (UniqueName: \"kubernetes.io/projected/d48c310f-8651-42ca-8d78-7344c29e26dd-kube-api-access-g22vh\") pod \"e2e-lifecycle-model-de210ed2-kserve-b94d584f9-vfl5v\" (UID: \"d48c310f-8651-42ca-8d78-7344c29e26dd\") " pod="ai-tenant-e2e-mt-de210ed2/e2e-lifecycle-model-de210ed2-kserve-b94d584f9-vfl5v" Jul 07 16:11:29.613085 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:11:29.613052 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d48c310f-8651-42ca-8d78-7344c29e26dd-tls-certs\") pod \"e2e-lifecycle-model-de210ed2-kserve-b94d584f9-vfl5v\" (UID: \"d48c310f-8651-42ca-8d78-7344c29e26dd\") " pod="ai-tenant-e2e-mt-de210ed2/e2e-lifecycle-model-de210ed2-kserve-b94d584f9-vfl5v" Jul 07 16:11:29.613289 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:11:29.613093 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d48c310f-8651-42ca-8d78-7344c29e26dd-home\") pod \"e2e-lifecycle-model-de210ed2-kserve-b94d584f9-vfl5v\" (UID: \"d48c310f-8651-42ca-8d78-7344c29e26dd\") " pod="ai-tenant-e2e-mt-de210ed2/e2e-lifecycle-model-de210ed2-kserve-b94d584f9-vfl5v" Jul 07 16:11:29.613289 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:11:29.613161 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d48c310f-8651-42ca-8d78-7344c29e26dd-model-cache\") pod \"e2e-lifecycle-model-de210ed2-kserve-b94d584f9-vfl5v\" (UID: \"d48c310f-8651-42ca-8d78-7344c29e26dd\") " pod="ai-tenant-e2e-mt-de210ed2/e2e-lifecycle-model-de210ed2-kserve-b94d584f9-vfl5v" Jul 07 16:11:29.613289 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:11:29.613197 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d48c310f-8651-42ca-8d78-7344c29e26dd-tmp-dir\") pod \"e2e-lifecycle-model-de210ed2-kserve-b94d584f9-vfl5v\" (UID: \"d48c310f-8651-42ca-8d78-7344c29e26dd\") " pod="ai-tenant-e2e-mt-de210ed2/e2e-lifecycle-model-de210ed2-kserve-b94d584f9-vfl5v" Jul 07 16:11:29.613289 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:11:29.613253 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d48c310f-8651-42ca-8d78-7344c29e26dd-dshm\") pod \"e2e-lifecycle-model-de210ed2-kserve-b94d584f9-vfl5v\" (UID: \"d48c310f-8651-42ca-8d78-7344c29e26dd\") " pod="ai-tenant-e2e-mt-de210ed2/e2e-lifecycle-model-de210ed2-kserve-b94d584f9-vfl5v" Jul 07 16:11:29.613289 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:11:29.613287 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g22vh\" (UniqueName: \"kubernetes.io/projected/d48c310f-8651-42ca-8d78-7344c29e26dd-kube-api-access-g22vh\") pod \"e2e-lifecycle-model-de210ed2-kserve-b94d584f9-vfl5v\" (UID: \"d48c310f-8651-42ca-8d78-7344c29e26dd\") " pod="ai-tenant-e2e-mt-de210ed2/e2e-lifecycle-model-de210ed2-kserve-b94d584f9-vfl5v" Jul 07 16:11:29.613633 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:11:29.613607 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d48c310f-8651-42ca-8d78-7344c29e26dd-home\") pod \"e2e-lifecycle-model-de210ed2-kserve-b94d584f9-vfl5v\" (UID: \"d48c310f-8651-42ca-8d78-7344c29e26dd\") " pod="ai-tenant-e2e-mt-de210ed2/e2e-lifecycle-model-de210ed2-kserve-b94d584f9-vfl5v" Jul 07 16:11:29.613693 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:11:29.613676 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d48c310f-8651-42ca-8d78-7344c29e26dd-tmp-dir\") pod \"e2e-lifecycle-model-de210ed2-kserve-b94d584f9-vfl5v\" (UID: \"d48c310f-8651-42ca-8d78-7344c29e26dd\") " pod="ai-tenant-e2e-mt-de210ed2/e2e-lifecycle-model-de210ed2-kserve-b94d584f9-vfl5v" Jul 07 16:11:29.613741 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:11:29.613702 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d48c310f-8651-42ca-8d78-7344c29e26dd-model-cache\") pod \"e2e-lifecycle-model-de210ed2-kserve-b94d584f9-vfl5v\" (UID: \"d48c310f-8651-42ca-8d78-7344c29e26dd\") " pod="ai-tenant-e2e-mt-de210ed2/e2e-lifecycle-model-de210ed2-kserve-b94d584f9-vfl5v" Jul 07 16:11:29.615464 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:11:29.615435 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d48c310f-8651-42ca-8d78-7344c29e26dd-dshm\") pod \"e2e-lifecycle-model-de210ed2-kserve-b94d584f9-vfl5v\" (UID: \"d48c310f-8651-42ca-8d78-7344c29e26dd\") " pod="ai-tenant-e2e-mt-de210ed2/e2e-lifecycle-model-de210ed2-kserve-b94d584f9-vfl5v" Jul 07 16:11:29.615636 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:11:29.615617 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d48c310f-8651-42ca-8d78-7344c29e26dd-tls-certs\") pod \"e2e-lifecycle-model-de210ed2-kserve-b94d584f9-vfl5v\" (UID: \"d48c310f-8651-42ca-8d78-7344c29e26dd\") " pod="ai-tenant-e2e-mt-de210ed2/e2e-lifecycle-model-de210ed2-kserve-b94d584f9-vfl5v" Jul 07 16:11:29.628659 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:11:29.628639 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g22vh\" (UniqueName: \"kubernetes.io/projected/d48c310f-8651-42ca-8d78-7344c29e26dd-kube-api-access-g22vh\") pod \"e2e-lifecycle-model-de210ed2-kserve-b94d584f9-vfl5v\" (UID: \"d48c310f-8651-42ca-8d78-7344c29e26dd\") " pod="ai-tenant-e2e-mt-de210ed2/e2e-lifecycle-model-de210ed2-kserve-b94d584f9-vfl5v" Jul 07 16:11:29.663325 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:11:29.663252 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-de210ed2-openshift-default-7b667c4c5f-k6cgm" Jul 07 16:11:29.663785 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:11:29.663573 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-de210ed2-openshift-default-7b667c4c5f-k6cgm" Jul 07 16:11:29.664190 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:11:29.664172 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-de210ed2-openshift-default-7b667c4c5f-k6cgm" Jul 07 16:11:29.713596 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:11:29.713563 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-de210ed2/e2e-lifecycle-model-de210ed2-kserve-b94d584f9-vfl5v" Jul 07 16:11:29.866275 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:11:29.866252 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-de210ed2/e2e-lifecycle-model-de210ed2-kserve-b94d584f9-vfl5v"] Jul 07 16:11:29.868439 ip-10-0-132-65 kubenswrapper[2565]: W0707 16:11:29.868414 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd48c310f_8651_42ca_8d78_7344c29e26dd.slice/crio-6d9c4af656791862b84e8e8e6e52c6af99894cb732e934e37e36ec31b3bdfc14 WatchSource:0}: Error finding container 6d9c4af656791862b84e8e8e6e52c6af99894cb732e934e37e36ec31b3bdfc14: Status 404 returned error can't find the container with id 6d9c4af656791862b84e8e8e6e52c6af99894cb732e934e37e36ec31b3bdfc14 Jul 07 16:11:30.311539 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:11:30.311485 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-de210ed2/e2e-lifecycle-model-de210ed2-kserve-b94d584f9-vfl5v" event={"ID":"d48c310f-8651-42ca-8d78-7344c29e26dd","Type":"ContainerStarted","Data":"51ed536268d6ac43036c1d4cd31cf97968bcac96c62cb727a27e7c462e9f89b5"} Jul 07 16:11:30.311715 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:11:30.311552 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-de210ed2/e2e-lifecycle-model-de210ed2-kserve-b94d584f9-vfl5v" event={"ID":"d48c310f-8651-42ca-8d78-7344c29e26dd","Type":"ContainerStarted","Data":"6d9c4af656791862b84e8e8e6e52c6af99894cb732e934e37e36ec31b3bdfc14"} Jul 07 16:11:30.336474 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:11:30.336427 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-de210ed2/e2e-lifecycle-model-de210ed2-kserve-b94d584f9-vfl5v" podStartSLOduration=1.336410933 podStartE2EDuration="1.336410933s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:11:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:11:30.335542832 +0000 UTC m=+1829.292120300" watchObservedRunningTime="2026-07-07 16:11:30.336410933 +0000 UTC m=+1829.292988400" Jul 07 16:11:39.714606 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:11:39.714566 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-de210ed2/e2e-lifecycle-model-de210ed2-kserve-b94d584f9-vfl5v" Jul 07 16:11:39.715094 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:11:39.714652 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-de210ed2/e2e-lifecycle-model-de210ed2-kserve-b94d584f9-vfl5v" Jul 07 16:11:39.727616 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:11:39.727590 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-de210ed2/e2e-lifecycle-model-de210ed2-kserve-b94d584f9-vfl5v" Jul 07 16:11:40.371655 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:11:40.371625 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-de210ed2/e2e-lifecycle-model-de210ed2-kserve-b94d584f9-vfl5v" Jul 07 16:11:58.625982 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:11:58.625951 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:11:59.088885 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:11:59.088852 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-mt-de210ed2-79fd54f7d8-b8nxq"] Jul 07 16:11:59.089087 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:11:59.089066 2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-mt-de210ed2-79fd54f7d8-b8nxq" podUID="1aefdb27-068e-4a46-9407-9baa5bb4182b" containerName="maas-api" containerID="cri-o://210b0c14b391e4adde3ea46661334ba965a5983966aa966a9d92e77a8e04a787" gracePeriod=30 Jul 07 16:11:59.111111 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:11:59.111086 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:11:59.224433 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:11:59.224390 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="opendatahub/maas-api-e2e-mt-de210ed2-79fd54f7d8-b8nxq" podUID="1aefdb27-068e-4a46-9407-9baa5bb4182b" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.57:8443/health\": dial tcp 10.133.0.57:8443: connect: connection refused" Jul 07 16:11:59.346201 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:11:59.346143 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-de210ed2-79fd54f7d8-b8nxq" Jul 07 16:11:59.400733 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:11:59.400693 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1aefdb27-068e-4a46-9407-9baa5bb4182b-openshift-service-ca\") pod \"1aefdb27-068e-4a46-9407-9baa5bb4182b\" (UID: \"1aefdb27-068e-4a46-9407-9baa5bb4182b\") " Jul 07 16:11:59.400912 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:11:59.400795 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1aefdb27-068e-4a46-9407-9baa5bb4182b-maas-api-tls\") pod \"1aefdb27-068e-4a46-9407-9baa5bb4182b\" (UID: \"1aefdb27-068e-4a46-9407-9baa5bb4182b\") " Jul 07 16:11:59.400912 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:11:59.400862 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gvn9b\" (UniqueName: \"kubernetes.io/projected/1aefdb27-068e-4a46-9407-9baa5bb4182b-kube-api-access-gvn9b\") pod \"1aefdb27-068e-4a46-9407-9baa5bb4182b\" (UID: \"1aefdb27-068e-4a46-9407-9baa5bb4182b\") " Jul 07 16:11:59.401100 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:11:59.401071 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1aefdb27-068e-4a46-9407-9baa5bb4182b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "1aefdb27-068e-4a46-9407-9baa5bb4182b" (UID: "1aefdb27-068e-4a46-9407-9baa5bb4182b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 16:11:59.401235 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:11:59.401215 2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1aefdb27-068e-4a46-9407-9baa5bb4182b-openshift-service-ca\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 07 16:11:59.402888 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:11:59.402865 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1aefdb27-068e-4a46-9407-9baa5bb4182b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "1aefdb27-068e-4a46-9407-9baa5bb4182b" (UID: "1aefdb27-068e-4a46-9407-9baa5bb4182b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 16:11:59.403001 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:11:59.402980 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1aefdb27-068e-4a46-9407-9baa5bb4182b-kube-api-access-gvn9b" (OuterVolumeSpecName: "kube-api-access-gvn9b") pod "1aefdb27-068e-4a46-9407-9baa5bb4182b" (UID: "1aefdb27-068e-4a46-9407-9baa5bb4182b"). InnerVolumeSpecName "kube-api-access-gvn9b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 16:11:59.453337 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:11:59.453303 2565 generic.go:356] "Generic (PLEG): container finished" podID="1aefdb27-068e-4a46-9407-9baa5bb4182b" containerID="210b0c14b391e4adde3ea46661334ba965a5983966aa966a9d92e77a8e04a787" exitCode=0 Jul 07 16:11:59.453479 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:11:59.453348 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-de210ed2-79fd54f7d8-b8nxq" Jul 07 16:11:59.453479 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:11:59.453386 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-de210ed2-79fd54f7d8-b8nxq" event={"ID":"1aefdb27-068e-4a46-9407-9baa5bb4182b","Type":"ContainerDied","Data":"210b0c14b391e4adde3ea46661334ba965a5983966aa966a9d92e77a8e04a787"} Jul 07 16:11:59.453479 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:11:59.453441 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-de210ed2-79fd54f7d8-b8nxq" event={"ID":"1aefdb27-068e-4a46-9407-9baa5bb4182b","Type":"ContainerDied","Data":"84c54e38504acf7fe91518858207d3664262f66c1165dc2fa2ff61a1abfe30a9"} Jul 07 16:11:59.453479 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:11:59.453464 2565 scope.go:117] "RemoveContainer" containerID="210b0c14b391e4adde3ea46661334ba965a5983966aa966a9d92e77a8e04a787" Jul 07 16:11:59.464563 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:11:59.464547 2565 scope.go:117] "RemoveContainer" containerID="210b0c14b391e4adde3ea46661334ba965a5983966aa966a9d92e77a8e04a787" Jul 07 16:11:59.464829 ip-10-0-132-65 kubenswrapper[2565]: E0707 16:11:59.464807 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"210b0c14b391e4adde3ea46661334ba965a5983966aa966a9d92e77a8e04a787\": container with ID starting with 210b0c14b391e4adde3ea46661334ba965a5983966aa966a9d92e77a8e04a787 not found: ID does not exist" containerID="210b0c14b391e4adde3ea46661334ba965a5983966aa966a9d92e77a8e04a787" Jul 07 16:11:59.464879 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:11:59.464838 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"210b0c14b391e4adde3ea46661334ba965a5983966aa966a9d92e77a8e04a787"} err="failed to get container status \"210b0c14b391e4adde3ea46661334ba965a5983966aa966a9d92e77a8e04a787\": rpc error: code = NotFound desc = could not find container \"210b0c14b391e4adde3ea46661334ba965a5983966aa966a9d92e77a8e04a787\": container with ID starting with 210b0c14b391e4adde3ea46661334ba965a5983966aa966a9d92e77a8e04a787 not found: ID does not exist" Jul 07 16:11:59.484798 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:11:59.484765 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-mt-de210ed2-79fd54f7d8-b8nxq"] Jul 07 16:11:59.487049 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:11:59.487024 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-mt-de210ed2-79fd54f7d8-b8nxq"] Jul 07 16:11:59.502354 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:11:59.502333 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gvn9b\" (UniqueName: \"kubernetes.io/projected/1aefdb27-068e-4a46-9407-9baa5bb4182b-kube-api-access-gvn9b\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 07 16:11:59.502437 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:11:59.502357 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1aefdb27-068e-4a46-9407-9baa5bb4182b-maas-api-tls\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 07 16:11:59.662155 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:11:59.662062 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1aefdb27-068e-4a46-9407-9baa5bb4182b" path="/var/lib/kubelet/pods/1aefdb27-068e-4a46-9407-9baa5bb4182b/volumes" Jul 07 16:12:04.701260 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:12:04.701224 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:12:09.694713 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:12:09.694683 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-de210ed2/e2e-lifecycle-model-de210ed2-kserve-b94d584f9-vfl5v"] Jul 07 16:12:09.794017 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:12:09.793986 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:12:24.715078 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:12:24.714984 2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-mt-de210ed2/e2e-lifecycle-model-de210ed2-kserve-b94d584f9-vfl5v" podUID="d48c310f-8651-42ca-8d78-7344c29e26dd" containerName="main" containerID="cri-o://51ed536268d6ac43036c1d4cd31cf97968bcac96c62cb727a27e7c462e9f89b5" gracePeriod=45 Jul 07 16:12:24.968060 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:12:24.968004 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-de210ed2/e2e-lifecycle-model-de210ed2-kserve-b94d584f9-vfl5v" Jul 07 16:12:25.031713 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:12:25.031687 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d48c310f-8651-42ca-8d78-7344c29e26dd-model-cache\") pod \"d48c310f-8651-42ca-8d78-7344c29e26dd\" (UID: \"d48c310f-8651-42ca-8d78-7344c29e26dd\") " Jul 07 16:12:25.031905 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:12:25.031726 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d48c310f-8651-42ca-8d78-7344c29e26dd-home\") pod \"d48c310f-8651-42ca-8d78-7344c29e26dd\" (UID: \"d48c310f-8651-42ca-8d78-7344c29e26dd\") " Jul 07 16:12:25.031905 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:12:25.031750 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d48c310f-8651-42ca-8d78-7344c29e26dd-tls-certs\") pod \"d48c310f-8651-42ca-8d78-7344c29e26dd\" (UID: \"d48c310f-8651-42ca-8d78-7344c29e26dd\") " Jul 07 16:12:25.031905 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:12:25.031786 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g22vh\" (UniqueName: \"kubernetes.io/projected/d48c310f-8651-42ca-8d78-7344c29e26dd-kube-api-access-g22vh\") pod \"d48c310f-8651-42ca-8d78-7344c29e26dd\" (UID: \"d48c310f-8651-42ca-8d78-7344c29e26dd\") " Jul 07 16:12:25.031905 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:12:25.031846 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d48c310f-8651-42ca-8d78-7344c29e26dd-tmp-dir\") pod \"d48c310f-8651-42ca-8d78-7344c29e26dd\" (UID: \"d48c310f-8651-42ca-8d78-7344c29e26dd\") " Jul 07 16:12:25.031905 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:12:25.031904 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d48c310f-8651-42ca-8d78-7344c29e26dd-dshm\") pod \"d48c310f-8651-42ca-8d78-7344c29e26dd\" (UID: \"d48c310f-8651-42ca-8d78-7344c29e26dd\") " Jul 07 16:12:25.032156 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:12:25.032058 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d48c310f-8651-42ca-8d78-7344c29e26dd-model-cache" (OuterVolumeSpecName: "model-cache") pod "d48c310f-8651-42ca-8d78-7344c29e26dd" (UID: "d48c310f-8651-42ca-8d78-7344c29e26dd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 16:12:25.032156 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:12:25.032048 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d48c310f-8651-42ca-8d78-7344c29e26dd-home" (OuterVolumeSpecName: "home") pod "d48c310f-8651-42ca-8d78-7344c29e26dd" (UID: "d48c310f-8651-42ca-8d78-7344c29e26dd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 16:12:25.032156 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:12:25.032111 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d48c310f-8651-42ca-8d78-7344c29e26dd-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d48c310f-8651-42ca-8d78-7344c29e26dd" (UID: "d48c310f-8651-42ca-8d78-7344c29e26dd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 16:12:25.032290 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:12:25.032255 2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d48c310f-8651-42ca-8d78-7344c29e26dd-model-cache\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 07 16:12:25.032290 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:12:25.032269 2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d48c310f-8651-42ca-8d78-7344c29e26dd-home\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 07 16:12:25.032290 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:12:25.032278 2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d48c310f-8651-42ca-8d78-7344c29e26dd-tmp-dir\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 07 16:12:25.034126 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:12:25.034101 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d48c310f-8651-42ca-8d78-7344c29e26dd-dshm" (OuterVolumeSpecName: "dshm") pod "d48c310f-8651-42ca-8d78-7344c29e26dd" (UID: "d48c310f-8651-42ca-8d78-7344c29e26dd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 16:12:25.034126 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:12:25.034113 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d48c310f-8651-42ca-8d78-7344c29e26dd-kube-api-access-g22vh" (OuterVolumeSpecName: "kube-api-access-g22vh") pod "d48c310f-8651-42ca-8d78-7344c29e26dd" (UID: "d48c310f-8651-42ca-8d78-7344c29e26dd"). InnerVolumeSpecName "kube-api-access-g22vh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 16:12:25.034295 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:12:25.034275 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d48c310f-8651-42ca-8d78-7344c29e26dd-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d48c310f-8651-42ca-8d78-7344c29e26dd" (UID: "d48c310f-8651-42ca-8d78-7344c29e26dd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 16:12:25.133449 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:12:25.133414 2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d48c310f-8651-42ca-8d78-7344c29e26dd-tls-certs\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 07 16:12:25.133449 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:12:25.133445 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g22vh\" (UniqueName: \"kubernetes.io/projected/d48c310f-8651-42ca-8d78-7344c29e26dd-kube-api-access-g22vh\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 07 16:12:25.133449 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:12:25.133454 2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d48c310f-8651-42ca-8d78-7344c29e26dd-dshm\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 07 16:12:25.584685 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:12:25.584654 2565 generic.go:356] "Generic (PLEG): container finished" podID="d48c310f-8651-42ca-8d78-7344c29e26dd" containerID="51ed536268d6ac43036c1d4cd31cf97968bcac96c62cb727a27e7c462e9f89b5" exitCode=0 Jul 07 16:12:25.584874 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:12:25.584695 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-de210ed2/e2e-lifecycle-model-de210ed2-kserve-b94d584f9-vfl5v" Jul 07 16:12:25.584874 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:12:25.584738 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-de210ed2/e2e-lifecycle-model-de210ed2-kserve-b94d584f9-vfl5v" event={"ID":"d48c310f-8651-42ca-8d78-7344c29e26dd","Type":"ContainerDied","Data":"51ed536268d6ac43036c1d4cd31cf97968bcac96c62cb727a27e7c462e9f89b5"} Jul 07 16:12:25.584874 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:12:25.584778 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-de210ed2/e2e-lifecycle-model-de210ed2-kserve-b94d584f9-vfl5v" event={"ID":"d48c310f-8651-42ca-8d78-7344c29e26dd","Type":"ContainerDied","Data":"6d9c4af656791862b84e8e8e6e52c6af99894cb732e934e37e36ec31b3bdfc14"} Jul 07 16:12:25.584874 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:12:25.584797 2565 scope.go:117] "RemoveContainer" containerID="51ed536268d6ac43036c1d4cd31cf97968bcac96c62cb727a27e7c462e9f89b5" Jul 07 16:12:25.593383 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:12:25.593369 2565 scope.go:117] "RemoveContainer" containerID="51ed536268d6ac43036c1d4cd31cf97968bcac96c62cb727a27e7c462e9f89b5" Jul 07 16:12:25.593650 ip-10-0-132-65 kubenswrapper[2565]: E0707 16:12:25.593633 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51ed536268d6ac43036c1d4cd31cf97968bcac96c62cb727a27e7c462e9f89b5\": container with ID starting with 51ed536268d6ac43036c1d4cd31cf97968bcac96c62cb727a27e7c462e9f89b5 not found: ID does not exist" containerID="51ed536268d6ac43036c1d4cd31cf97968bcac96c62cb727a27e7c462e9f89b5" Jul 07 16:12:25.593728 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:12:25.593659 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51ed536268d6ac43036c1d4cd31cf97968bcac96c62cb727a27e7c462e9f89b5"} err="failed to get container status \"51ed536268d6ac43036c1d4cd31cf97968bcac96c62cb727a27e7c462e9f89b5\": rpc error: code = NotFound desc = could not find container \"51ed536268d6ac43036c1d4cd31cf97968bcac96c62cb727a27e7c462e9f89b5\": container with ID starting with 51ed536268d6ac43036c1d4cd31cf97968bcac96c62cb727a27e7c462e9f89b5 not found: ID does not exist" Jul 07 16:12:25.611120 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:12:25.611093 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-de210ed2/e2e-lifecycle-model-de210ed2-kserve-b94d584f9-vfl5v"] Jul 07 16:12:25.615467 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:12:25.615442 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-de210ed2/e2e-lifecycle-model-de210ed2-kserve-b94d584f9-vfl5v"] Jul 07 16:12:25.661370 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:12:25.661338 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d48c310f-8651-42ca-8d78-7344c29e26dd" path="/var/lib/kubelet/pods/d48c310f-8651-42ca-8d78-7344c29e26dd/volumes" Jul 07 16:12:55.986311 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:12:55.986274 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:12:57.102651 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:12:57.102613 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:13:03.709735 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:13:03.709695 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:13:03.789036 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:13:03.789006 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:13:04.008920 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:13:04.008840 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-de210ed2-openshift-default-7b667c4c5f-k6cgm"] Jul 07 16:13:04.009070 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:13:04.009050 2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-de210ed2-openshift-default-7b667c4c5f-k6cgm" podUID="40a5a7ce-85bb-43f3-b33e-9af689d3e95c" containerName="istio-proxy" containerID="cri-o://b0ee929e8a63b5dc5be950f3aaa06bda27924763ead674ec32e6272df0cc4567" gracePeriod=30 Jul 07 16:13:04.024773 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:13:04.024744 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:13:04.282666 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:13:04.282579 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:13:04.900703 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:13:04.900671 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:13:05.420028 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:13:05.419988 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:13:05.523834 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:13:05.523782 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:13:05.625664 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:13:05.625631 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:13:06.458432 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:13:06.458403 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:13:06.558016 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:13:06.557987 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:13:06.657051 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:13:06.657018 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:13:08.250846 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:13:08.250816 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:13:09.300421 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:13:09.300396 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-de210ed2-openshift-default-7b667c4c5f-k6cgm" Jul 07 16:13:09.452897 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:13:09.452858 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40a5a7ce-85bb-43f3-b33e-9af689d3e95c-istiod-ca-cert\") pod \"40a5a7ce-85bb-43f3-b33e-9af689d3e95c\" (UID: \"40a5a7ce-85bb-43f3-b33e-9af689d3e95c\") " Jul 07 16:13:09.453093 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:13:09.452909 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40a5a7ce-85bb-43f3-b33e-9af689d3e95c-credential-socket\") pod \"40a5a7ce-85bb-43f3-b33e-9af689d3e95c\" (UID: \"40a5a7ce-85bb-43f3-b33e-9af689d3e95c\") " Jul 07 16:13:09.453093 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:13:09.452964 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xsm5q\" (UniqueName: \"kubernetes.io/projected/40a5a7ce-85bb-43f3-b33e-9af689d3e95c-kube-api-access-xsm5q\") pod \"40a5a7ce-85bb-43f3-b33e-9af689d3e95c\" (UID: \"40a5a7ce-85bb-43f3-b33e-9af689d3e95c\") " Jul 07 16:13:09.453093 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:13:09.453001 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40a5a7ce-85bb-43f3-b33e-9af689d3e95c-istio-podinfo\") pod \"40a5a7ce-85bb-43f3-b33e-9af689d3e95c\" (UID: \"40a5a7ce-85bb-43f3-b33e-9af689d3e95c\") " Jul 07 16:13:09.453093 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:13:09.453059 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40a5a7ce-85bb-43f3-b33e-9af689d3e95c-workload-socket\") pod \"40a5a7ce-85bb-43f3-b33e-9af689d3e95c\" (UID: \"40a5a7ce-85bb-43f3-b33e-9af689d3e95c\") " Jul 07 16:13:09.453093 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:13:09.453091 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40a5a7ce-85bb-43f3-b33e-9af689d3e95c-istio-data\") pod \"40a5a7ce-85bb-43f3-b33e-9af689d3e95c\" (UID: \"40a5a7ce-85bb-43f3-b33e-9af689d3e95c\") " Jul 07 16:13:09.453361 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:13:09.453159 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40a5a7ce-85bb-43f3-b33e-9af689d3e95c-istio-envoy\") pod \"40a5a7ce-85bb-43f3-b33e-9af689d3e95c\" (UID: \"40a5a7ce-85bb-43f3-b33e-9af689d3e95c\") " Jul 07 16:13:09.453361 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:13:09.453180 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40a5a7ce-85bb-43f3-b33e-9af689d3e95c-istio-token\") pod \"40a5a7ce-85bb-43f3-b33e-9af689d3e95c\" (UID: \"40a5a7ce-85bb-43f3-b33e-9af689d3e95c\") " Jul 07 16:13:09.453361 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:13:09.453224 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40a5a7ce-85bb-43f3-b33e-9af689d3e95c-workload-certs\") pod \"40a5a7ce-85bb-43f3-b33e-9af689d3e95c\" (UID: \"40a5a7ce-85bb-43f3-b33e-9af689d3e95c\") " Jul 07 16:13:09.453361 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:13:09.453234 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40a5a7ce-85bb-43f3-b33e-9af689d3e95c-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "40a5a7ce-85bb-43f3-b33e-9af689d3e95c" (UID: "40a5a7ce-85bb-43f3-b33e-9af689d3e95c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 16:13:09.453361 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:13:09.453274 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40a5a7ce-85bb-43f3-b33e-9af689d3e95c-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "40a5a7ce-85bb-43f3-b33e-9af689d3e95c" (UID: "40a5a7ce-85bb-43f3-b33e-9af689d3e95c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 16:13:09.453705 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:13:09.453606 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40a5a7ce-85bb-43f3-b33e-9af689d3e95c-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "40a5a7ce-85bb-43f3-b33e-9af689d3e95c" (UID: "40a5a7ce-85bb-43f3-b33e-9af689d3e95c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 16:13:09.453843 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:13:09.453811 2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40a5a7ce-85bb-43f3-b33e-9af689d3e95c-workload-certs\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 07 16:13:09.453843 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:13:09.453835 2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40a5a7ce-85bb-43f3-b33e-9af689d3e95c-istiod-ca-cert\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 07 16:13:09.453971 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:13:09.453847 2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40a5a7ce-85bb-43f3-b33e-9af689d3e95c-credential-socket\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 07 16:13:09.454127 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:13:09.454097 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40a5a7ce-85bb-43f3-b33e-9af689d3e95c-istio-data" (OuterVolumeSpecName: "istio-data") pod "40a5a7ce-85bb-43f3-b33e-9af689d3e95c" (UID: "40a5a7ce-85bb-43f3-b33e-9af689d3e95c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 16:13:09.454203 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:13:09.454172 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40a5a7ce-85bb-43f3-b33e-9af689d3e95c-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "40a5a7ce-85bb-43f3-b33e-9af689d3e95c" (UID: "40a5a7ce-85bb-43f3-b33e-9af689d3e95c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 16:13:09.455356 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:13:09.455333 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40a5a7ce-85bb-43f3-b33e-9af689d3e95c-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "40a5a7ce-85bb-43f3-b33e-9af689d3e95c" (UID: "40a5a7ce-85bb-43f3-b33e-9af689d3e95c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 16:13:09.455465 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:13:09.455374 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40a5a7ce-85bb-43f3-b33e-9af689d3e95c-istio-token" (OuterVolumeSpecName: "istio-token") pod "40a5a7ce-85bb-43f3-b33e-9af689d3e95c" (UID: "40a5a7ce-85bb-43f3-b33e-9af689d3e95c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 16:13:09.455541 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:13:09.455459 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40a5a7ce-85bb-43f3-b33e-9af689d3e95c-kube-api-access-xsm5q" (OuterVolumeSpecName: "kube-api-access-xsm5q") pod "40a5a7ce-85bb-43f3-b33e-9af689d3e95c" (UID: "40a5a7ce-85bb-43f3-b33e-9af689d3e95c"). InnerVolumeSpecName "kube-api-access-xsm5q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 16:13:09.455670 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:13:09.455646 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/40a5a7ce-85bb-43f3-b33e-9af689d3e95c-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "40a5a7ce-85bb-43f3-b33e-9af689d3e95c" (UID: "40a5a7ce-85bb-43f3-b33e-9af689d3e95c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 07 16:13:09.555336 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:13:09.555244 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xsm5q\" (UniqueName: \"kubernetes.io/projected/40a5a7ce-85bb-43f3-b33e-9af689d3e95c-kube-api-access-xsm5q\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 07 16:13:09.555336 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:13:09.555281 2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40a5a7ce-85bb-43f3-b33e-9af689d3e95c-istio-podinfo\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 07 16:13:09.555336 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:13:09.555290 2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40a5a7ce-85bb-43f3-b33e-9af689d3e95c-workload-socket\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 07 16:13:09.555336 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:13:09.555299 2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40a5a7ce-85bb-43f3-b33e-9af689d3e95c-istio-data\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 07 16:13:09.555336 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:13:09.555308 2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40a5a7ce-85bb-43f3-b33e-9af689d3e95c-istio-envoy\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 07 16:13:09.555336 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:13:09.555315 2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40a5a7ce-85bb-43f3-b33e-9af689d3e95c-istio-token\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 07 16:13:09.805625 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:13:09.805536 2565 generic.go:356] "Generic (PLEG): container finished" podID="40a5a7ce-85bb-43f3-b33e-9af689d3e95c" containerID="b0ee929e8a63b5dc5be950f3aaa06bda27924763ead674ec32e6272df0cc4567" exitCode=0 Jul 07 16:13:09.805625 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:13:09.805585 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-de210ed2-openshift-default-7b667c4c5f-k6cgm" Jul 07 16:13:09.805625 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:13:09.805613 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-de210ed2-openshift-default-7b667c4c5f-k6cgm" event={"ID":"40a5a7ce-85bb-43f3-b33e-9af689d3e95c","Type":"ContainerDied","Data":"b0ee929e8a63b5dc5be950f3aaa06bda27924763ead674ec32e6272df0cc4567"} Jul 07 16:13:09.805875 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:13:09.805672 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-de210ed2-openshift-default-7b667c4c5f-k6cgm" event={"ID":"40a5a7ce-85bb-43f3-b33e-9af689d3e95c","Type":"ContainerDied","Data":"aab690cd0df8f5ae912041078c8d123d23cba4b2b8d282b3ce3089bcbb1a9f15"} Jul 07 16:13:09.805875 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:13:09.805704 2565 scope.go:117] "RemoveContainer" containerID="b0ee929e8a63b5dc5be950f3aaa06bda27924763ead674ec32e6272df0cc4567" Jul 07 16:13:09.814876 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:13:09.814863 2565 scope.go:117] "RemoveContainer" containerID="1ebcd6eb0e8c4f77b9c7a0b479d6507c204c9e7c3ed2564c09be9c86d45c346b" Jul 07 16:13:09.823066 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:13:09.823044 2565 scope.go:117] "RemoveContainer" containerID="b0ee929e8a63b5dc5be950f3aaa06bda27924763ead674ec32e6272df0cc4567" Jul 07 16:13:09.823376 ip-10-0-132-65 kubenswrapper[2565]: E0707 16:13:09.823340 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0ee929e8a63b5dc5be950f3aaa06bda27924763ead674ec32e6272df0cc4567\": container with ID starting with b0ee929e8a63b5dc5be950f3aaa06bda27924763ead674ec32e6272df0cc4567 not found: ID does not exist" containerID="b0ee929e8a63b5dc5be950f3aaa06bda27924763ead674ec32e6272df0cc4567" Jul 07 16:13:09.823485 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:13:09.823387 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0ee929e8a63b5dc5be950f3aaa06bda27924763ead674ec32e6272df0cc4567"} err="failed to get container status \"b0ee929e8a63b5dc5be950f3aaa06bda27924763ead674ec32e6272df0cc4567\": rpc error: code = NotFound desc = could not find container \"b0ee929e8a63b5dc5be950f3aaa06bda27924763ead674ec32e6272df0cc4567\": container with ID starting with b0ee929e8a63b5dc5be950f3aaa06bda27924763ead674ec32e6272df0cc4567 not found: ID does not exist" Jul 07 16:13:09.823485 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:13:09.823412 2565 scope.go:117] "RemoveContainer" containerID="1ebcd6eb0e8c4f77b9c7a0b479d6507c204c9e7c3ed2564c09be9c86d45c346b" Jul 07 16:13:09.823762 ip-10-0-132-65 kubenswrapper[2565]: E0707 16:13:09.823738 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ebcd6eb0e8c4f77b9c7a0b479d6507c204c9e7c3ed2564c09be9c86d45c346b\": container with ID starting with 1ebcd6eb0e8c4f77b9c7a0b479d6507c204c9e7c3ed2564c09be9c86d45c346b not found: ID does not exist" containerID="1ebcd6eb0e8c4f77b9c7a0b479d6507c204c9e7c3ed2564c09be9c86d45c346b" Jul 07 16:13:09.823815 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:13:09.823769 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ebcd6eb0e8c4f77b9c7a0b479d6507c204c9e7c3ed2564c09be9c86d45c346b"} err="failed to get container status \"1ebcd6eb0e8c4f77b9c7a0b479d6507c204c9e7c3ed2564c09be9c86d45c346b\": rpc error: code = NotFound desc = could not find container \"1ebcd6eb0e8c4f77b9c7a0b479d6507c204c9e7c3ed2564c09be9c86d45c346b\": container with ID starting with 1ebcd6eb0e8c4f77b9c7a0b479d6507c204c9e7c3ed2564c09be9c86d45c346b not found: ID does not exist" Jul 07 16:13:09.839972 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:13:09.839930 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-de210ed2-openshift-default-7b667c4c5f-k6cgm"] Jul 07 16:13:09.843000 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:13:09.842977 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-de210ed2-openshift-default-7b667c4c5f-k6cgm"] Jul 07 16:13:11.660212 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:13:11.660179 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="40a5a7ce-85bb-43f3-b33e-9af689d3e95c" path="/var/lib/kubelet/pods/40a5a7ce-85bb-43f3-b33e-9af689d3e95c/volumes" Jul 07 16:13:15.086349 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:13:15.086318 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:13:15.164051 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:13:15.164022 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:13:15.834637 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:13:15.834599 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:13:22.886885 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:13:22.886850 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:13:24.607143 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:13:24.607106 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:13:33.662672 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:13:33.662640 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:13:34.062717 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:13:34.062690 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:13:55.073951 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:13:55.073872 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:14:01.769935 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:14:01.769894 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:14:01.874915 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:14:01.874879 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:14:02.174485 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:14:02.174453 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:14:02.661973 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:14:02.661932 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-10313d-openshift-default-697cf4bfbf-q5zcm"] Jul 07 16:14:02.662534 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:14:02.662516 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="40a5a7ce-85bb-43f3-b33e-9af689d3e95c" containerName="istio-proxy" Jul 07 16:14:02.662587 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:14:02.662538 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="40a5a7ce-85bb-43f3-b33e-9af689d3e95c" containerName="istio-proxy" Jul 07 16:14:02.662587 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:14:02.662555 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d48c310f-8651-42ca-8d78-7344c29e26dd" containerName="main" Jul 07 16:14:02.662587 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:14:02.662565 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="d48c310f-8651-42ca-8d78-7344c29e26dd" containerName="main" Jul 07 16:14:02.662587 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:14:02.662584 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1aefdb27-068e-4a46-9407-9baa5bb4182b" containerName="maas-api" Jul 07 16:14:02.662716 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:14:02.662594 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="1aefdb27-068e-4a46-9407-9baa5bb4182b" containerName="maas-api" Jul 07 16:14:02.662716 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:14:02.662626 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="40a5a7ce-85bb-43f3-b33e-9af689d3e95c" containerName="istio-proxy" Jul 07 16:14:02.662716 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:14:02.662633 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="40a5a7ce-85bb-43f3-b33e-9af689d3e95c" containerName="istio-proxy" Jul 07 16:14:02.662823 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:14:02.662718 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="40a5a7ce-85bb-43f3-b33e-9af689d3e95c" containerName="istio-proxy" Jul 07 16:14:02.662823 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:14:02.662728 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="d48c310f-8651-42ca-8d78-7344c29e26dd" containerName="main" Jul 07 16:14:02.662823 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:14:02.662740 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="40a5a7ce-85bb-43f3-b33e-9af689d3e95c" containerName="istio-proxy" Jul 07 16:14:02.662823 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:14:02.662751 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="1aefdb27-068e-4a46-9407-9baa5bb4182b" containerName="maas-api" Jul 07 16:14:02.667444 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:14:02.667429 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-10313d-openshift-default-697cf4bfbf-q5zcm" Jul 07 16:14:02.670629 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:14:02.670614 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-10313d-openshift-default-dockercfg-k778v\"" Jul 07 16:14:02.685001 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:14:02.684964 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-10313d-openshift-default-697cf4bfbf-q5zcm"] Jul 07 16:14:02.707611 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:14:02.707583 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:14:02.832214 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:14:02.832181 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8zwj\" (UniqueName: \"kubernetes.io/projected/c0d8c8f4-4a61-4842-bdb2-d9ad60f7c833-kube-api-access-z8zwj\") pod \"e2e-inf-a-10313d-openshift-default-697cf4bfbf-q5zcm\" (UID: \"c0d8c8f4-4a61-4842-bdb2-d9ad60f7c833\") " pod="openshift-ingress/e2e-inf-a-10313d-openshift-default-697cf4bfbf-q5zcm" Jul 07 16:14:02.832664 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:14:02.832233 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0d8c8f4-4a61-4842-bdb2-d9ad60f7c833-istiod-ca-cert\") pod \"e2e-inf-a-10313d-openshift-default-697cf4bfbf-q5zcm\" (UID: \"c0d8c8f4-4a61-4842-bdb2-d9ad60f7c833\") " pod="openshift-ingress/e2e-inf-a-10313d-openshift-default-697cf4bfbf-q5zcm" Jul 07 16:14:02.832664 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:14:02.832259 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0d8c8f4-4a61-4842-bdb2-d9ad60f7c833-workload-certs\") pod \"e2e-inf-a-10313d-openshift-default-697cf4bfbf-q5zcm\" (UID: \"c0d8c8f4-4a61-4842-bdb2-d9ad60f7c833\") " pod="openshift-ingress/e2e-inf-a-10313d-openshift-default-697cf4bfbf-q5zcm" Jul 07 16:14:02.832664 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:14:02.832304 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0d8c8f4-4a61-4842-bdb2-d9ad60f7c833-credential-socket\") pod \"e2e-inf-a-10313d-openshift-default-697cf4bfbf-q5zcm\" (UID: \"c0d8c8f4-4a61-4842-bdb2-d9ad60f7c833\") " pod="openshift-ingress/e2e-inf-a-10313d-openshift-default-697cf4bfbf-q5zcm" Jul 07 16:14:02.832664 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:14:02.832321 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0d8c8f4-4a61-4842-bdb2-d9ad60f7c833-workload-socket\") pod \"e2e-inf-a-10313d-openshift-default-697cf4bfbf-q5zcm\" (UID: \"c0d8c8f4-4a61-4842-bdb2-d9ad60f7c833\") " pod="openshift-ingress/e2e-inf-a-10313d-openshift-default-697cf4bfbf-q5zcm" Jul 07 16:14:02.832664 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:14:02.832340 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0d8c8f4-4a61-4842-bdb2-d9ad60f7c833-istio-token\") pod \"e2e-inf-a-10313d-openshift-default-697cf4bfbf-q5zcm\" (UID: \"c0d8c8f4-4a61-4842-bdb2-d9ad60f7c833\") " pod="openshift-ingress/e2e-inf-a-10313d-openshift-default-697cf4bfbf-q5zcm" Jul 07 16:14:02.832664 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:14:02.832359 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0d8c8f4-4a61-4842-bdb2-d9ad60f7c833-istio-podinfo\") pod \"e2e-inf-a-10313d-openshift-default-697cf4bfbf-q5zcm\" (UID: \"c0d8c8f4-4a61-4842-bdb2-d9ad60f7c833\") " pod="openshift-ingress/e2e-inf-a-10313d-openshift-default-697cf4bfbf-q5zcm" Jul 07 16:14:02.832664 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:14:02.832427 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0d8c8f4-4a61-4842-bdb2-d9ad60f7c833-istio-data\") pod \"e2e-inf-a-10313d-openshift-default-697cf4bfbf-q5zcm\" (UID: \"c0d8c8f4-4a61-4842-bdb2-d9ad60f7c833\") " pod="openshift-ingress/e2e-inf-a-10313d-openshift-default-697cf4bfbf-q5zcm" Jul 07 16:14:02.832664 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:14:02.832464 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0d8c8f4-4a61-4842-bdb2-d9ad60f7c833-istio-envoy\") pod \"e2e-inf-a-10313d-openshift-default-697cf4bfbf-q5zcm\" (UID: \"c0d8c8f4-4a61-4842-bdb2-d9ad60f7c833\") " pod="openshift-ingress/e2e-inf-a-10313d-openshift-default-697cf4bfbf-q5zcm" Jul 07 16:14:02.933169 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:14:02.933087 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0d8c8f4-4a61-4842-bdb2-d9ad60f7c833-istio-podinfo\") pod \"e2e-inf-a-10313d-openshift-default-697cf4bfbf-q5zcm\" (UID: \"c0d8c8f4-4a61-4842-bdb2-d9ad60f7c833\") " pod="openshift-ingress/e2e-inf-a-10313d-openshift-default-697cf4bfbf-q5zcm" Jul 07 16:14:02.933169 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:14:02.933139 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0d8c8f4-4a61-4842-bdb2-d9ad60f7c833-istio-data\") pod \"e2e-inf-a-10313d-openshift-default-697cf4bfbf-q5zcm\" (UID: \"c0d8c8f4-4a61-4842-bdb2-d9ad60f7c833\") " pod="openshift-ingress/e2e-inf-a-10313d-openshift-default-697cf4bfbf-q5zcm" Jul 07 16:14:02.933360 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:14:02.933172 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0d8c8f4-4a61-4842-bdb2-d9ad60f7c833-istio-envoy\") pod \"e2e-inf-a-10313d-openshift-default-697cf4bfbf-q5zcm\" (UID: \"c0d8c8f4-4a61-4842-bdb2-d9ad60f7c833\") " pod="openshift-ingress/e2e-inf-a-10313d-openshift-default-697cf4bfbf-q5zcm" Jul 07 16:14:02.933360 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:14:02.933211 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8zwj\" (UniqueName: \"kubernetes.io/projected/c0d8c8f4-4a61-4842-bdb2-d9ad60f7c833-kube-api-access-z8zwj\") pod \"e2e-inf-a-10313d-openshift-default-697cf4bfbf-q5zcm\" (UID: \"c0d8c8f4-4a61-4842-bdb2-d9ad60f7c833\") " pod="openshift-ingress/e2e-inf-a-10313d-openshift-default-697cf4bfbf-q5zcm" Jul 07 16:14:02.933360 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:14:02.933262 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0d8c8f4-4a61-4842-bdb2-d9ad60f7c833-istiod-ca-cert\") pod \"e2e-inf-a-10313d-openshift-default-697cf4bfbf-q5zcm\" (UID: \"c0d8c8f4-4a61-4842-bdb2-d9ad60f7c833\") " pod="openshift-ingress/e2e-inf-a-10313d-openshift-default-697cf4bfbf-q5zcm" Jul 07 16:14:02.933360 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:14:02.933295 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0d8c8f4-4a61-4842-bdb2-d9ad60f7c833-workload-certs\") pod \"e2e-inf-a-10313d-openshift-default-697cf4bfbf-q5zcm\" (UID: \"c0d8c8f4-4a61-4842-bdb2-d9ad60f7c833\") " pod="openshift-ingress/e2e-inf-a-10313d-openshift-default-697cf4bfbf-q5zcm" Jul 07 16:14:02.933360 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:14:02.933346 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0d8c8f4-4a61-4842-bdb2-d9ad60f7c833-credential-socket\") pod \"e2e-inf-a-10313d-openshift-default-697cf4bfbf-q5zcm\" (UID: \"c0d8c8f4-4a61-4842-bdb2-d9ad60f7c833\") " pod="openshift-ingress/e2e-inf-a-10313d-openshift-default-697cf4bfbf-q5zcm" Jul 07 16:14:02.933660 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:14:02.933369 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0d8c8f4-4a61-4842-bdb2-d9ad60f7c833-workload-socket\") pod \"e2e-inf-a-10313d-openshift-default-697cf4bfbf-q5zcm\" (UID: \"c0d8c8f4-4a61-4842-bdb2-d9ad60f7c833\") " pod="openshift-ingress/e2e-inf-a-10313d-openshift-default-697cf4bfbf-q5zcm" Jul 07 16:14:02.933660 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:14:02.933403 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0d8c8f4-4a61-4842-bdb2-d9ad60f7c833-istio-token\") pod \"e2e-inf-a-10313d-openshift-default-697cf4bfbf-q5zcm\" (UID: \"c0d8c8f4-4a61-4842-bdb2-d9ad60f7c833\") " pod="openshift-ingress/e2e-inf-a-10313d-openshift-default-697cf4bfbf-q5zcm" Jul 07 16:14:02.933767 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:14:02.933673 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0d8c8f4-4a61-4842-bdb2-d9ad60f7c833-istio-data\") pod \"e2e-inf-a-10313d-openshift-default-697cf4bfbf-q5zcm\" (UID: \"c0d8c8f4-4a61-4842-bdb2-d9ad60f7c833\") " pod="openshift-ingress/e2e-inf-a-10313d-openshift-default-697cf4bfbf-q5zcm" Jul 07 16:14:02.933833 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:14:02.933799 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0d8c8f4-4a61-4842-bdb2-d9ad60f7c833-workload-certs\") pod \"e2e-inf-a-10313d-openshift-default-697cf4bfbf-q5zcm\" (UID: \"c0d8c8f4-4a61-4842-bdb2-d9ad60f7c833\") " pod="openshift-ingress/e2e-inf-a-10313d-openshift-default-697cf4bfbf-q5zcm" Jul 07 16:14:02.933880 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:14:02.933822 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0d8c8f4-4a61-4842-bdb2-d9ad60f7c833-credential-socket\") pod \"e2e-inf-a-10313d-openshift-default-697cf4bfbf-q5zcm\" (UID: \"c0d8c8f4-4a61-4842-bdb2-d9ad60f7c833\") " pod="openshift-ingress/e2e-inf-a-10313d-openshift-default-697cf4bfbf-q5zcm" Jul 07 16:14:02.933924 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:14:02.933894 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0d8c8f4-4a61-4842-bdb2-d9ad60f7c833-workload-socket\") pod \"e2e-inf-a-10313d-openshift-default-697cf4bfbf-q5zcm\" (UID: \"c0d8c8f4-4a61-4842-bdb2-d9ad60f7c833\") " pod="openshift-ingress/e2e-inf-a-10313d-openshift-default-697cf4bfbf-q5zcm" Jul 07 16:14:02.934084 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:14:02.934059 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0d8c8f4-4a61-4842-bdb2-d9ad60f7c833-istiod-ca-cert\") pod \"e2e-inf-a-10313d-openshift-default-697cf4bfbf-q5zcm\" (UID: \"c0d8c8f4-4a61-4842-bdb2-d9ad60f7c833\") " pod="openshift-ingress/e2e-inf-a-10313d-openshift-default-697cf4bfbf-q5zcm" Jul 07 16:14:02.935256 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:14:02.935232 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0d8c8f4-4a61-4842-bdb2-d9ad60f7c833-istio-envoy\") pod \"e2e-inf-a-10313d-openshift-default-697cf4bfbf-q5zcm\" (UID: \"c0d8c8f4-4a61-4842-bdb2-d9ad60f7c833\") " pod="openshift-ingress/e2e-inf-a-10313d-openshift-default-697cf4bfbf-q5zcm" Jul 07 16:14:02.936012 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:14:02.935990 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0d8c8f4-4a61-4842-bdb2-d9ad60f7c833-istio-podinfo\") pod \"e2e-inf-a-10313d-openshift-default-697cf4bfbf-q5zcm\" (UID: \"c0d8c8f4-4a61-4842-bdb2-d9ad60f7c833\") " pod="openshift-ingress/e2e-inf-a-10313d-openshift-default-697cf4bfbf-q5zcm" Jul 07 16:14:02.956783 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:14:02.956754 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8zwj\" (UniqueName: \"kubernetes.io/projected/c0d8c8f4-4a61-4842-bdb2-d9ad60f7c833-kube-api-access-z8zwj\") pod \"e2e-inf-a-10313d-openshift-default-697cf4bfbf-q5zcm\" (UID: \"c0d8c8f4-4a61-4842-bdb2-d9ad60f7c833\") " pod="openshift-ingress/e2e-inf-a-10313d-openshift-default-697cf4bfbf-q5zcm" Jul 07 16:14:02.957457 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:14:02.957436 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0d8c8f4-4a61-4842-bdb2-d9ad60f7c833-istio-token\") pod \"e2e-inf-a-10313d-openshift-default-697cf4bfbf-q5zcm\" (UID: \"c0d8c8f4-4a61-4842-bdb2-d9ad60f7c833\") " pod="openshift-ingress/e2e-inf-a-10313d-openshift-default-697cf4bfbf-q5zcm" Jul 07 16:14:02.977652 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:14:02.977626 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-10313d-openshift-default-697cf4bfbf-q5zcm" Jul 07 16:14:02.999319 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:14:02.999287 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:14:03.124380 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:14:03.124346 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-10313d-openshift-default-697cf4bfbf-q5zcm"] Jul 07 16:14:03.127534 ip-10-0-132-65 kubenswrapper[2565]: W0707 16:14:03.127509 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0d8c8f4_4a61_4842_bdb2_d9ad60f7c833.slice/crio-e3a160ac5fefce446c098888a8a444806231841148fb63fc81258e882f38f8ca WatchSource:0}: Error finding container e3a160ac5fefce446c098888a8a444806231841148fb63fc81258e882f38f8ca: Status 404 returned error can't find the container with id e3a160ac5fefce446c098888a8a444806231841148fb63fc81258e882f38f8ca Jul 07 16:14:03.129637 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:14:03.129605 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 07 16:14:03.129729 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:14:03.129666 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 07 16:14:03.129729 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:14:03.129694 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 07 16:14:04.069455 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:14:04.069418 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-10313d-openshift-default-697cf4bfbf-q5zcm" event={"ID":"c0d8c8f4-4a61-4842-bdb2-d9ad60f7c833","Type":"ContainerStarted","Data":"0f27bf5e2618b699a1e42e79e429a79d1383fa5288753cee1d50659f6f85351e"} Jul 07 16:14:04.069872 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:14:04.069476 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-10313d-openshift-default-697cf4bfbf-q5zcm" event={"ID":"c0d8c8f4-4a61-4842-bdb2-d9ad60f7c833","Type":"ContainerStarted","Data":"e3a160ac5fefce446c098888a8a444806231841148fb63fc81258e882f38f8ca"} Jul 07 16:14:04.100579 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:14:04.100524 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-10313d-openshift-default-697cf4bfbf-q5zcm" podStartSLOduration=2.100507108 podStartE2EDuration="2.100507108s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:14:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:14:04.100028448 +0000 UTC m=+1983.056605915" watchObservedRunningTime="2026-07-07 16:14:04.100507108 +0000 UTC m=+1983.057084571" Jul 07 16:14:04.978022 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:14:04.977988 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-10313d-openshift-default-697cf4bfbf-q5zcm" Jul 07 16:14:04.982955 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:14:04.982928 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-10313d-openshift-default-697cf4bfbf-q5zcm" Jul 07 16:14:05.073911 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:14:05.073877 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-10313d-openshift-default-697cf4bfbf-q5zcm" Jul 07 16:14:05.074842 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:14:05.074820 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-10313d-openshift-default-697cf4bfbf-q5zcm" Jul 07 16:14:06.107870 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:14:06.107826 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:14:06.176435 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:14:06.176405 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-inf-a-10313d-7c69d759f7-fdz4j"] Jul 07 16:14:06.180085 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:14:06.180063 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-a-10313d-7c69d759f7-fdz4j" Jul 07 16:14:06.183099 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:14:06.183081 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-inf-a-10313d-serving-cert\"" Jul 07 16:14:06.183221 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:14:06.183111 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-dockercfg-9bqq6\"" Jul 07 16:14:06.192172 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:14:06.192150 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-inf-a-10313d-7c69d759f7-fdz4j"] Jul 07 16:14:06.222331 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:14:06.222304 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:14:06.266988 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:14:06.266958 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zffpp\" (UniqueName: \"kubernetes.io/projected/dca7b67e-1b6f-4572-861c-4a22277d5100-kube-api-access-zffpp\") pod \"maas-api-e2e-inf-a-10313d-7c69d759f7-fdz4j\" (UID: \"dca7b67e-1b6f-4572-861c-4a22277d5100\") " pod="opendatahub/maas-api-e2e-inf-a-10313d-7c69d759f7-fdz4j" Jul 07 16:14:06.267194 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:14:06.267012 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dca7b67e-1b6f-4572-861c-4a22277d5100-openshift-service-ca\") pod \"maas-api-e2e-inf-a-10313d-7c69d759f7-fdz4j\" (UID: \"dca7b67e-1b6f-4572-861c-4a22277d5100\") " pod="opendatahub/maas-api-e2e-inf-a-10313d-7c69d759f7-fdz4j" Jul 07 16:14:06.267194 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:14:06.267147 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dca7b67e-1b6f-4572-861c-4a22277d5100-maas-api-tls\") pod \"maas-api-e2e-inf-a-10313d-7c69d759f7-fdz4j\" (UID: \"dca7b67e-1b6f-4572-861c-4a22277d5100\") " pod="opendatahub/maas-api-e2e-inf-a-10313d-7c69d759f7-fdz4j" Jul 07 16:14:06.322313 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:14:06.322276 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:14:06.367808 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:14:06.367723 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dca7b67e-1b6f-4572-861c-4a22277d5100-maas-api-tls\") pod \"maas-api-e2e-inf-a-10313d-7c69d759f7-fdz4j\" (UID: \"dca7b67e-1b6f-4572-861c-4a22277d5100\") " pod="opendatahub/maas-api-e2e-inf-a-10313d-7c69d759f7-fdz4j" Jul 07 16:14:06.367808 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:14:06.367775 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zffpp\" (UniqueName: \"kubernetes.io/projected/dca7b67e-1b6f-4572-861c-4a22277d5100-kube-api-access-zffpp\") pod \"maas-api-e2e-inf-a-10313d-7c69d759f7-fdz4j\" (UID: \"dca7b67e-1b6f-4572-861c-4a22277d5100\") " pod="opendatahub/maas-api-e2e-inf-a-10313d-7c69d759f7-fdz4j" Jul 07 16:14:06.367997 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:14:06.367826 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dca7b67e-1b6f-4572-861c-4a22277d5100-openshift-service-ca\") pod \"maas-api-e2e-inf-a-10313d-7c69d759f7-fdz4j\" (UID: \"dca7b67e-1b6f-4572-861c-4a22277d5100\") " pod="opendatahub/maas-api-e2e-inf-a-10313d-7c69d759f7-fdz4j" Jul 07 16:14:06.368538 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:14:06.368520 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dca7b67e-1b6f-4572-861c-4a22277d5100-openshift-service-ca\") pod \"maas-api-e2e-inf-a-10313d-7c69d759f7-fdz4j\" (UID: \"dca7b67e-1b6f-4572-861c-4a22277d5100\") " pod="opendatahub/maas-api-e2e-inf-a-10313d-7c69d759f7-fdz4j" Jul 07 16:14:06.370055 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:14:06.370033 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dca7b67e-1b6f-4572-861c-4a22277d5100-maas-api-tls\") pod \"maas-api-e2e-inf-a-10313d-7c69d759f7-fdz4j\" (UID: \"dca7b67e-1b6f-4572-861c-4a22277d5100\") " pod="opendatahub/maas-api-e2e-inf-a-10313d-7c69d759f7-fdz4j" Jul 07 16:14:06.378006 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:14:06.377985 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zffpp\" (UniqueName: \"kubernetes.io/projected/dca7b67e-1b6f-4572-861c-4a22277d5100-kube-api-access-zffpp\") pod \"maas-api-e2e-inf-a-10313d-7c69d759f7-fdz4j\" (UID: \"dca7b67e-1b6f-4572-861c-4a22277d5100\") " pod="opendatahub/maas-api-e2e-inf-a-10313d-7c69d759f7-fdz4j" Jul 07 16:14:06.420721 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:14:06.420689 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:14:06.492832 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:14:06.492798 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-a-10313d-7c69d759f7-fdz4j" Jul 07 16:14:06.641337 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:14:06.641260 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-inf-a-10313d-7c69d759f7-fdz4j"] Jul 07 16:14:06.644365 ip-10-0-132-65 kubenswrapper[2565]: W0707 16:14:06.644335 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddca7b67e_1b6f_4572_861c_4a22277d5100.slice/crio-ba34581190e2e339e5c680fc32cc315030566b7dc4aead3b2746b0cfefc1db7a WatchSource:0}: Error finding container ba34581190e2e339e5c680fc32cc315030566b7dc4aead3b2746b0cfefc1db7a: Status 404 returned error can't find the container with id ba34581190e2e339e5c680fc32cc315030566b7dc4aead3b2746b0cfefc1db7a Jul 07 16:14:06.645855 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:14:06.645829 2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 07 16:14:06.741741 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:14:06.741706 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:14:06.847991 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:14:06.847957 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:14:07.085292 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:14:07.085253 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-a-10313d-7c69d759f7-fdz4j" event={"ID":"dca7b67e-1b6f-4572-861c-4a22277d5100","Type":"ContainerStarted","Data":"ba34581190e2e339e5c680fc32cc315030566b7dc4aead3b2746b0cfefc1db7a"} Jul 07 16:14:08.091141 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:14:08.091095 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-a-10313d-7c69d759f7-fdz4j" event={"ID":"dca7b67e-1b6f-4572-861c-4a22277d5100","Type":"ContainerStarted","Data":"7a04fb85927fb70b20490254f512e76da7862829dc472f61acca928a8f0ccb47"} Jul 07 16:14:08.091577 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:14:08.091218 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-inf-a-10313d-7c69d759f7-fdz4j" Jul 07 16:14:08.113133 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:14:08.113081 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-inf-a-10313d-7c69d759f7-fdz4j" podStartSLOduration=1.595191515 podStartE2EDuration="2.11306384s" totalImagesPullingTime="517.872325ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:14:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:14:08.111237897 +0000 UTC m=+1987.067815378" watchObservedRunningTime="2026-07-07 16:14:08.11306384 +0000 UTC m=+1987.069641306" Jul 07 16:14:10.652223 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:14:10.652192 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:14:11.000154 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:14:11.000075 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:14:11.314833 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:14:11.314807 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-bda481-openshift-default-849c85bbf5-2jnt7"] Jul 07 16:14:11.318231 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:14:11.318210 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-bda481-openshift-default-849c85bbf5-2jnt7" Jul 07 16:14:11.321735 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:14:11.321716 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-bda481-openshift-default-dockercfg-5c9p6\"" Jul 07 16:14:11.332984 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:14:11.332962 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-bda481-openshift-default-849c85bbf5-2jnt7"] Jul 07 16:14:11.365605 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:14:11.365549 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:14:11.415181 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:14:11.415145 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrknk\" (UniqueName: \"kubernetes.io/projected/13c0936e-d449-4b9b-882e-587993a0df6f-kube-api-access-xrknk\") pod \"e2e-inf-b-bda481-openshift-default-849c85bbf5-2jnt7\" (UID: \"13c0936e-d449-4b9b-882e-587993a0df6f\") " pod="openshift-ingress/e2e-inf-b-bda481-openshift-default-849c85bbf5-2jnt7" Jul 07 16:14:11.415181 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:14:11.415195 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/13c0936e-d449-4b9b-882e-587993a0df6f-istio-token\") pod \"e2e-inf-b-bda481-openshift-default-849c85bbf5-2jnt7\" (UID: \"13c0936e-d449-4b9b-882e-587993a0df6f\") " pod="openshift-ingress/e2e-inf-b-bda481-openshift-default-849c85bbf5-2jnt7" Jul 07 16:14:11.415420 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:14:11.415237 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/13c0936e-d449-4b9b-882e-587993a0df6f-istiod-ca-cert\") pod \"e2e-inf-b-bda481-openshift-default-849c85bbf5-2jnt7\" (UID: \"13c0936e-d449-4b9b-882e-587993a0df6f\") " pod="openshift-ingress/e2e-inf-b-bda481-openshift-default-849c85bbf5-2jnt7" Jul 07 16:14:11.415420 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:14:11.415284 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/13c0936e-d449-4b9b-882e-587993a0df6f-istio-envoy\") pod \"e2e-inf-b-bda481-openshift-default-849c85bbf5-2jnt7\" (UID: \"13c0936e-d449-4b9b-882e-587993a0df6f\") " pod="openshift-ingress/e2e-inf-b-bda481-openshift-default-849c85bbf5-2jnt7" Jul 07 16:14:11.415420 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:14:11.415346 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/13c0936e-d449-4b9b-882e-587993a0df6f-credential-socket\") pod \"e2e-inf-b-bda481-openshift-default-849c85bbf5-2jnt7\" (UID: \"13c0936e-d449-4b9b-882e-587993a0df6f\") " pod="openshift-ingress/e2e-inf-b-bda481-openshift-default-849c85bbf5-2jnt7" Jul 07 16:14:11.415420 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:14:11.415383 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/13c0936e-d449-4b9b-882e-587993a0df6f-workload-certs\") pod \"e2e-inf-b-bda481-openshift-default-849c85bbf5-2jnt7\" (UID: \"13c0936e-d449-4b9b-882e-587993a0df6f\") " pod="openshift-ingress/e2e-inf-b-bda481-openshift-default-849c85bbf5-2jnt7" Jul 07 16:14:11.415609 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:14:11.415437 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/13c0936e-d449-4b9b-882e-587993a0df6f-istio-data\") pod \"e2e-inf-b-bda481-openshift-default-849c85bbf5-2jnt7\" (UID: \"13c0936e-d449-4b9b-882e-587993a0df6f\") " pod="openshift-ingress/e2e-inf-b-bda481-openshift-default-849c85bbf5-2jnt7" Jul 07 16:14:11.415609 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:14:11.415457 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/13c0936e-d449-4b9b-882e-587993a0df6f-workload-socket\") pod \"e2e-inf-b-bda481-openshift-default-849c85bbf5-2jnt7\" (UID: \"13c0936e-d449-4b9b-882e-587993a0df6f\") " pod="openshift-ingress/e2e-inf-b-bda481-openshift-default-849c85bbf5-2jnt7" Jul 07 16:14:11.415609 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:14:11.415542 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/13c0936e-d449-4b9b-882e-587993a0df6f-istio-podinfo\") pod \"e2e-inf-b-bda481-openshift-default-849c85bbf5-2jnt7\" (UID: \"13c0936e-d449-4b9b-882e-587993a0df6f\") " pod="openshift-ingress/e2e-inf-b-bda481-openshift-default-849c85bbf5-2jnt7" Jul 07 16:14:11.508363 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:14:11.508329 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:14:11.516600 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:14:11.516575 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/13c0936e-d449-4b9b-882e-587993a0df6f-istio-envoy\") pod \"e2e-inf-b-bda481-openshift-default-849c85bbf5-2jnt7\" (UID: \"13c0936e-d449-4b9b-882e-587993a0df6f\") " pod="openshift-ingress/e2e-inf-b-bda481-openshift-default-849c85bbf5-2jnt7" Jul 07 16:14:11.516688 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:14:11.516629 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/13c0936e-d449-4b9b-882e-587993a0df6f-credential-socket\") pod \"e2e-inf-b-bda481-openshift-default-849c85bbf5-2jnt7\" (UID: \"13c0936e-d449-4b9b-882e-587993a0df6f\") " pod="openshift-ingress/e2e-inf-b-bda481-openshift-default-849c85bbf5-2jnt7" Jul 07 16:14:11.516688 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:14:11.516661 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/13c0936e-d449-4b9b-882e-587993a0df6f-workload-certs\") pod \"e2e-inf-b-bda481-openshift-default-849c85bbf5-2jnt7\" (UID: \"13c0936e-d449-4b9b-882e-587993a0df6f\") " pod="openshift-ingress/e2e-inf-b-bda481-openshift-default-849c85bbf5-2jnt7" Jul 07 16:14:11.516805 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:14:11.516703 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/13c0936e-d449-4b9b-882e-587993a0df6f-istio-data\") pod \"e2e-inf-b-bda481-openshift-default-849c85bbf5-2jnt7\" (UID: \"13c0936e-d449-4b9b-882e-587993a0df6f\") " pod="openshift-ingress/e2e-inf-b-bda481-openshift-default-849c85bbf5-2jnt7" Jul 07 16:14:11.516805 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:14:11.516725 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/13c0936e-d449-4b9b-882e-587993a0df6f-workload-socket\") pod \"e2e-inf-b-bda481-openshift-default-849c85bbf5-2jnt7\" (UID: \"13c0936e-d449-4b9b-882e-587993a0df6f\") " pod="openshift-ingress/e2e-inf-b-bda481-openshift-default-849c85bbf5-2jnt7" Jul 07 16:14:11.516805 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:14:11.516791 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/13c0936e-d449-4b9b-882e-587993a0df6f-istio-podinfo\") pod \"e2e-inf-b-bda481-openshift-default-849c85bbf5-2jnt7\" (UID: \"13c0936e-d449-4b9b-882e-587993a0df6f\") " pod="openshift-ingress/e2e-inf-b-bda481-openshift-default-849c85bbf5-2jnt7" Jul 07 16:14:11.516961 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:14:11.516817 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrknk\" (UniqueName: \"kubernetes.io/projected/13c0936e-d449-4b9b-882e-587993a0df6f-kube-api-access-xrknk\") pod \"e2e-inf-b-bda481-openshift-default-849c85bbf5-2jnt7\" (UID: \"13c0936e-d449-4b9b-882e-587993a0df6f\") " pod="openshift-ingress/e2e-inf-b-bda481-openshift-default-849c85bbf5-2jnt7" Jul 07 16:14:11.516961 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:14:11.516840 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/13c0936e-d449-4b9b-882e-587993a0df6f-istio-token\") pod \"e2e-inf-b-bda481-openshift-default-849c85bbf5-2jnt7\" (UID: \"13c0936e-d449-4b9b-882e-587993a0df6f\") " pod="openshift-ingress/e2e-inf-b-bda481-openshift-default-849c85bbf5-2jnt7" Jul 07 16:14:11.516961 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:14:11.516855 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/13c0936e-d449-4b9b-882e-587993a0df6f-istiod-ca-cert\") pod \"e2e-inf-b-bda481-openshift-default-849c85bbf5-2jnt7\" (UID: \"13c0936e-d449-4b9b-882e-587993a0df6f\") " pod="openshift-ingress/e2e-inf-b-bda481-openshift-default-849c85bbf5-2jnt7" Jul 07 16:14:11.517134 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:14:11.517102 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/13c0936e-d449-4b9b-882e-587993a0df6f-credential-socket\") pod \"e2e-inf-b-bda481-openshift-default-849c85bbf5-2jnt7\" (UID: \"13c0936e-d449-4b9b-882e-587993a0df6f\") " pod="openshift-ingress/e2e-inf-b-bda481-openshift-default-849c85bbf5-2jnt7" Jul 07 16:14:11.517606 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:14:11.517572 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/13c0936e-d449-4b9b-882e-587993a0df6f-istiod-ca-cert\") pod \"e2e-inf-b-bda481-openshift-default-849c85bbf5-2jnt7\" (UID: \"13c0936e-d449-4b9b-882e-587993a0df6f\") " pod="openshift-ingress/e2e-inf-b-bda481-openshift-default-849c85bbf5-2jnt7" Jul 07 16:14:11.517606 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:14:11.517590 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/13c0936e-d449-4b9b-882e-587993a0df6f-istio-data\") pod \"e2e-inf-b-bda481-openshift-default-849c85bbf5-2jnt7\" (UID: \"13c0936e-d449-4b9b-882e-587993a0df6f\") " pod="openshift-ingress/e2e-inf-b-bda481-openshift-default-849c85bbf5-2jnt7" Jul 07 16:14:11.517774 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:14:11.517572 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/13c0936e-d449-4b9b-882e-587993a0df6f-workload-certs\") pod \"e2e-inf-b-bda481-openshift-default-849c85bbf5-2jnt7\" (UID: \"13c0936e-d449-4b9b-882e-587993a0df6f\") " pod="openshift-ingress/e2e-inf-b-bda481-openshift-default-849c85bbf5-2jnt7" Jul 07 16:14:11.517774 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:14:11.517685 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/13c0936e-d449-4b9b-882e-587993a0df6f-workload-socket\") pod \"e2e-inf-b-bda481-openshift-default-849c85bbf5-2jnt7\" (UID: \"13c0936e-d449-4b9b-882e-587993a0df6f\") " pod="openshift-ingress/e2e-inf-b-bda481-openshift-default-849c85bbf5-2jnt7" Jul 07 16:14:11.518975 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:14:11.518953 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/13c0936e-d449-4b9b-882e-587993a0df6f-istio-envoy\") pod \"e2e-inf-b-bda481-openshift-default-849c85bbf5-2jnt7\" (UID: \"13c0936e-d449-4b9b-882e-587993a0df6f\") " pod="openshift-ingress/e2e-inf-b-bda481-openshift-default-849c85bbf5-2jnt7" Jul 07 16:14:11.519256 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:14:11.519237 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/13c0936e-d449-4b9b-882e-587993a0df6f-istio-podinfo\") pod \"e2e-inf-b-bda481-openshift-default-849c85bbf5-2jnt7\" (UID: \"13c0936e-d449-4b9b-882e-587993a0df6f\") " pod="openshift-ingress/e2e-inf-b-bda481-openshift-default-849c85bbf5-2jnt7" Jul 07 16:14:11.527952 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:14:11.527933 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/13c0936e-d449-4b9b-882e-587993a0df6f-istio-token\") pod \"e2e-inf-b-bda481-openshift-default-849c85bbf5-2jnt7\" (UID: \"13c0936e-d449-4b9b-882e-587993a0df6f\") " pod="openshift-ingress/e2e-inf-b-bda481-openshift-default-849c85bbf5-2jnt7" Jul 07 16:14:11.529454 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:14:11.529434 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrknk\" (UniqueName: \"kubernetes.io/projected/13c0936e-d449-4b9b-882e-587993a0df6f-kube-api-access-xrknk\") pod \"e2e-inf-b-bda481-openshift-default-849c85bbf5-2jnt7\" (UID: \"13c0936e-d449-4b9b-882e-587993a0df6f\") " pod="openshift-ingress/e2e-inf-b-bda481-openshift-default-849c85bbf5-2jnt7" Jul 07 16:14:11.629527 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:14:11.629410 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-bda481-openshift-default-849c85bbf5-2jnt7" Jul 07 16:14:11.779345 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:14:11.779319 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-bda481-openshift-default-849c85bbf5-2jnt7"] Jul 07 16:14:11.780661 ip-10-0-132-65 kubenswrapper[2565]: W0707 16:14:11.780630 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13c0936e_d449_4b9b_882e_587993a0df6f.slice/crio-b59afbfd78d58516193b53d5f60de198ce75f7ab06d9e0d0747ba652b6b8a0dd WatchSource:0}: Error finding container b59afbfd78d58516193b53d5f60de198ce75f7ab06d9e0d0747ba652b6b8a0dd: Status 404 returned error can't find the container with id b59afbfd78d58516193b53d5f60de198ce75f7ab06d9e0d0747ba652b6b8a0dd Jul 07 16:14:11.782878 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:14:11.782847 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 07 16:14:11.782958 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:14:11.782911 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 07 16:14:11.782958 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:14:11.782939 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 07 16:14:12.110419 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:14:12.110379 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-bda481-openshift-default-849c85bbf5-2jnt7" event={"ID":"13c0936e-d449-4b9b-882e-587993a0df6f","Type":"ContainerStarted","Data":"b5954bc8b67181ea56e930a6330aa2c88c7ab359c69efdfb25a4ac6915c37309"} Jul 07 16:14:12.110618 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:14:12.110436 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-bda481-openshift-default-849c85bbf5-2jnt7" event={"ID":"13c0936e-d449-4b9b-882e-587993a0df6f","Type":"ContainerStarted","Data":"b59afbfd78d58516193b53d5f60de198ce75f7ab06d9e0d0747ba652b6b8a0dd"} Jul 07 16:14:12.140342 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:14:12.140297 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-bda481-openshift-default-849c85bbf5-2jnt7" podStartSLOduration=1.140280241 podStartE2EDuration="1.140280241s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:14:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:14:12.137457111 +0000 UTC m=+1991.094034580" watchObservedRunningTime="2026-07-07 16:14:12.140280241 +0000 UTC m=+1991.096857710" Jul 07 16:14:12.629908 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:14:12.629872 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-bda481-openshift-default-849c85bbf5-2jnt7" Jul 07 16:14:12.634388 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:14:12.634356 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-bda481-openshift-default-849c85bbf5-2jnt7" Jul 07 16:14:13.115238 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:14:13.115212 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-bda481-openshift-default-849c85bbf5-2jnt7" Jul 07 16:14:13.116244 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:14:13.116223 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-bda481-openshift-default-849c85bbf5-2jnt7" Jul 07 16:14:14.101656 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:14:14.101630 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-inf-a-10313d-7c69d759f7-fdz4j" Jul 07 16:14:14.170423 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:14:14.170386 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:14:14.283540 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:14:14.283508 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:14:14.379056 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:14:14.378930 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:14:14.773613 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:14:14.773463 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:14:14.916421 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:14:14.916390 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:14:15.021873 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:14:15.021842 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:14:18.673104 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:14:18.673030 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:14:20.076267 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:14:20.076111 2565 patch_prober.go:28] interesting pod/e2e-inf-a-10313d-openshift-default-697cf4bfbf-q5zcm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Jul 07 16:14:20.076267 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:14:20.076194 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-10313d-openshift-default-697cf4bfbf-q5zcm" podUID="c0d8c8f4-4a61-4842-bdb2-d9ad60f7c833" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Jul 07 16:14:20.153137 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:14:20.153055 2565 generic.go:356] "Generic (PLEG): container finished" podID="c0d8c8f4-4a61-4842-bdb2-d9ad60f7c833" containerID="0f27bf5e2618b699a1e42e79e429a79d1383fa5288753cee1d50659f6f85351e" exitCode=0 Jul 07 16:14:20.153137 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:14:20.153114 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-10313d-openshift-default-697cf4bfbf-q5zcm" event={"ID":"c0d8c8f4-4a61-4842-bdb2-d9ad60f7c833","Type":"ContainerDied","Data":"0f27bf5e2618b699a1e42e79e429a79d1383fa5288753cee1d50659f6f85351e"} Jul 07 16:14:20.153571 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:14:20.153555 2565 scope.go:117] "RemoveContainer" containerID="0f27bf5e2618b699a1e42e79e429a79d1383fa5288753cee1d50659f6f85351e" Jul 07 16:14:20.154668 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:14:20.154636 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 07 16:14:20.154735 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:14:20.154720 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 07 16:14:20.154786 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:14:20.154768 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 07 16:14:21.160248 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:14:21.160203 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-10313d-openshift-default-697cf4bfbf-q5zcm" event={"ID":"c0d8c8f4-4a61-4842-bdb2-d9ad60f7c833","Type":"ContainerStarted","Data":"b8f281a419118677bbe348c7af6f6175431cda33b410c7441922b8aeecdf6cae"} Jul 07 16:14:21.978826 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:14:21.978781 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-10313d-openshift-default-697cf4bfbf-q5zcm" Jul 07 16:14:21.979052 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:14:21.978911 2565 patch_prober.go:28] interesting pod/e2e-inf-a-10313d-openshift-default-697cf4bfbf-q5zcm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Jul 07 16:14:21.979052 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:14:21.978992 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-10313d-openshift-default-697cf4bfbf-q5zcm" podUID="c0d8c8f4-4a61-4842-bdb2-d9ad60f7c833" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Jul 07 16:14:22.979170 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:14:22.978689 2565 patch_prober.go:28] interesting pod/e2e-inf-a-10313d-openshift-default-697cf4bfbf-q5zcm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Jul 07 16:14:22.979170 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:14:22.978771 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-10313d-openshift-default-697cf4bfbf-q5zcm" podUID="c0d8c8f4-4a61-4842-bdb2-d9ad60f7c833" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Jul 07 16:14:23.978938 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:14:23.978900 2565 patch_prober.go:28] interesting pod/e2e-inf-a-10313d-openshift-default-697cf4bfbf-q5zcm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Jul 07 16:14:23.979234 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:14:23.979208 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-10313d-openshift-default-697cf4bfbf-q5zcm" podUID="c0d8c8f4-4a61-4842-bdb2-d9ad60f7c833" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Jul 07 16:14:24.978275 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:14:24.978229 2565 patch_prober.go:28] interesting pod/e2e-inf-a-10313d-openshift-default-697cf4bfbf-q5zcm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Jul 07 16:14:24.978510 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:14:24.978301 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-10313d-openshift-default-697cf4bfbf-q5zcm" podUID="c0d8c8f4-4a61-4842-bdb2-d9ad60f7c833" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Jul 07 16:14:25.983250 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:14:25.983216 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-10313d-openshift-default-697cf4bfbf-q5zcm" Jul 07 16:14:25.983673 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:14:25.983502 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-10313d-openshift-default-697cf4bfbf-q5zcm" Jul 07 16:14:25.984079 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:14:25.984063 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-10313d-openshift-default-697cf4bfbf-q5zcm" Jul 07 16:14:26.194473 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:14:26.194441 2565 generic.go:356] "Generic (PLEG): container finished" podID="13c0936e-d449-4b9b-882e-587993a0df6f" containerID="b5954bc8b67181ea56e930a6330aa2c88c7ab359c69efdfb25a4ac6915c37309" exitCode=0 Jul 07 16:14:26.194665 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:14:26.194517 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-bda481-openshift-default-849c85bbf5-2jnt7" event={"ID":"13c0936e-d449-4b9b-882e-587993a0df6f","Type":"ContainerDied","Data":"b5954bc8b67181ea56e930a6330aa2c88c7ab359c69efdfb25a4ac6915c37309"} Jul 07 16:14:26.195002 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:14:26.194986 2565 scope.go:117] "RemoveContainer" containerID="b5954bc8b67181ea56e930a6330aa2c88c7ab359c69efdfb25a4ac6915c37309" Jul 07 16:14:26.195940 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:14:26.195918 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 07 16:14:26.195992 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:14:26.195979 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 07 16:14:26.196028 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:14:26.196011 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 07 16:14:27.201833 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:14:27.201792 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-bda481-openshift-default-849c85bbf5-2jnt7" event={"ID":"13c0936e-d449-4b9b-882e-587993a0df6f","Type":"ContainerStarted","Data":"2423ee310d70d2fc3dccdd53c697a553aeea8db977eec3e5c6756ff88cca77e8"} Jul 07 16:14:27.629682 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:14:27.629641 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-bda481-openshift-default-849c85bbf5-2jnt7" Jul 07 16:14:27.630536 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:14:27.630510 2565 patch_prober.go:28] interesting pod/e2e-inf-b-bda481-openshift-default-849c85bbf5-2jnt7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Jul 07 16:14:27.630613 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:14:27.630568 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-bda481-openshift-default-849c85bbf5-2jnt7" podUID="13c0936e-d449-4b9b-882e-587993a0df6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Jul 07 16:14:28.116121 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:14:28.116089 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-bda481-openshift-default-849c85bbf5-2jnt7" Jul 07 16:14:28.615091 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:14:28.615056 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:14:28.630481 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:14:28.630450 2565 patch_prober.go:28] interesting pod/e2e-inf-b-bda481-openshift-default-849c85bbf5-2jnt7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Jul 07 16:14:28.630645 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:14:28.630532 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-bda481-openshift-default-849c85bbf5-2jnt7" podUID="13c0936e-d449-4b9b-882e-587993a0df6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Jul 07 16:14:28.715666 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:14:28.715631 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:14:29.629802 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:14:29.629763 2565 patch_prober.go:28] interesting pod/e2e-inf-b-bda481-openshift-default-849c85bbf5-2jnt7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Jul 07 16:14:29.630229 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:14:29.629823 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-bda481-openshift-default-849c85bbf5-2jnt7" podUID="13c0936e-d449-4b9b-882e-587993a0df6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Jul 07 16:14:30.629850 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:14:30.629816 2565 patch_prober.go:28] interesting pod/e2e-inf-b-bda481-openshift-default-849c85bbf5-2jnt7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body= Jul 07 16:14:30.630235 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:14:30.629869 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-bda481-openshift-default-849c85bbf5-2jnt7" podUID="13c0936e-d449-4b9b-882e-587993a0df6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" Jul 07 16:14:31.634108 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:14:31.634076 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-bda481-openshift-default-849c85bbf5-2jnt7" Jul 07 16:14:31.635103 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:14:31.635083 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-bda481-openshift-default-849c85bbf5-2jnt7" Jul 07 16:14:35.858370 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:14:35.858328 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:14:35.957740 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:14:35.957680 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:14:42.176453 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:14:42.176420 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:15:11.342770 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:15:11.342740 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:15:14.629215 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:15:14.629180 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:15:37.981141 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:15:37.981045 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:15:39.020677 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:15:39.020642 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:15:39.347241 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:15:39.347207 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:15:39.780639 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:15:39.780552 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:15:39.922004 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:15:39.921972 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:15:40.021544 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:15:40.021474 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:15:40.168606 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:15:40.168574 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-inf-a-10313d-7c69d759f7-fdz4j"] Jul 07 16:15:40.168819 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:15:40.168794 2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-inf-a-10313d-7c69d759f7-fdz4j" podUID="dca7b67e-1b6f-4572-861c-4a22277d5100" containerName="maas-api" containerID="cri-o://7a04fb85927fb70b20490254f512e76da7862829dc472f61acca928a8f0ccb47" gracePeriod=30 Jul 07 16:15:40.210657 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:15:40.210627 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:15:40.572021 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:15:40.571989 2565 generic.go:356] "Generic (PLEG): container finished" podID="dca7b67e-1b6f-4572-861c-4a22277d5100" containerID="7a04fb85927fb70b20490254f512e76da7862829dc472f61acca928a8f0ccb47" exitCode=0 Jul 07 16:15:40.572214 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:15:40.572073 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-a-10313d-7c69d759f7-fdz4j" event={"ID":"dca7b67e-1b6f-4572-861c-4a22277d5100","Type":"ContainerDied","Data":"7a04fb85927fb70b20490254f512e76da7862829dc472f61acca928a8f0ccb47"} Jul 07 16:15:40.618478 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:15:40.618455 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-a-10313d-7c69d759f7-fdz4j" Jul 07 16:15:40.695507 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:15:40.695469 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dca7b67e-1b6f-4572-861c-4a22277d5100-maas-api-tls\") pod \"dca7b67e-1b6f-4572-861c-4a22277d5100\" (UID: \"dca7b67e-1b6f-4572-861c-4a22277d5100\") " Jul 07 16:15:40.695692 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:15:40.695545 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zffpp\" (UniqueName: \"kubernetes.io/projected/dca7b67e-1b6f-4572-861c-4a22277d5100-kube-api-access-zffpp\") pod \"dca7b67e-1b6f-4572-861c-4a22277d5100\" (UID: \"dca7b67e-1b6f-4572-861c-4a22277d5100\") " Jul 07 16:15:40.695692 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:15:40.695579 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dca7b67e-1b6f-4572-861c-4a22277d5100-openshift-service-ca\") pod \"dca7b67e-1b6f-4572-861c-4a22277d5100\" (UID: \"dca7b67e-1b6f-4572-861c-4a22277d5100\") " Jul 07 16:15:40.696007 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:15:40.695981 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dca7b67e-1b6f-4572-861c-4a22277d5100-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "dca7b67e-1b6f-4572-861c-4a22277d5100" (UID: "dca7b67e-1b6f-4572-861c-4a22277d5100"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 16:15:40.697517 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:15:40.697473 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dca7b67e-1b6f-4572-861c-4a22277d5100-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "dca7b67e-1b6f-4572-861c-4a22277d5100" (UID: "dca7b67e-1b6f-4572-861c-4a22277d5100"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 16:15:40.697627 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:15:40.697609 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dca7b67e-1b6f-4572-861c-4a22277d5100-kube-api-access-zffpp" (OuterVolumeSpecName: "kube-api-access-zffpp") pod "dca7b67e-1b6f-4572-861c-4a22277d5100" (UID: "dca7b67e-1b6f-4572-861c-4a22277d5100"). InnerVolumeSpecName "kube-api-access-zffpp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 16:15:40.796331 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:15:40.796293 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dca7b67e-1b6f-4572-861c-4a22277d5100-maas-api-tls\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 07 16:15:40.796331 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:15:40.796325 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zffpp\" (UniqueName: \"kubernetes.io/projected/dca7b67e-1b6f-4572-861c-4a22277d5100-kube-api-access-zffpp\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 07 16:15:40.796331 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:15:40.796338 2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dca7b67e-1b6f-4572-861c-4a22277d5100-openshift-service-ca\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 07 16:15:41.578354 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:15:41.578325 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-a-10313d-7c69d759f7-fdz4j" Jul 07 16:15:41.578850 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:15:41.578366 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-a-10313d-7c69d759f7-fdz4j" event={"ID":"dca7b67e-1b6f-4572-861c-4a22277d5100","Type":"ContainerDied","Data":"ba34581190e2e339e5c680fc32cc315030566b7dc4aead3b2746b0cfefc1db7a"} Jul 07 16:15:41.578850 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:15:41.578444 2565 scope.go:117] "RemoveContainer" containerID="7a04fb85927fb70b20490254f512e76da7862829dc472f61acca928a8f0ccb47" Jul 07 16:15:41.604388 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:15:41.604362 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-inf-a-10313d-7c69d759f7-fdz4j"] Jul 07 16:15:41.606316 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:15:41.606283 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-inf-a-10313d-7c69d759f7-fdz4j"] Jul 07 16:15:41.662192 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:15:41.662161 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dca7b67e-1b6f-4572-861c-4a22277d5100" path="/var/lib/kubelet/pods/dca7b67e-1b6f-4572-861c-4a22277d5100/volumes" Jul 07 16:15:44.943719 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:15:44.943672 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:15:53.561026 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:15:53.560994 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:15:53.659991 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:15:53.659961 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:16:01.665185 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:01.665159 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-s9kl8_62da0165-d32c-4bba-9844-a49154d765e5/console-operator/1.log" Jul 07 16:16:01.667221 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:01.667202 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-s9kl8_62da0165-d32c-4bba-9844-a49154d765e5/console-operator/1.log" Jul 07 16:16:01.669648 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:01.669627 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-942xh_31ff16f0-6054-4eba-942a-b07a5a7afac7/ovn-acl-logging/0.log" Jul 07 16:16:01.671824 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:01.671806 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-942xh_31ff16f0-6054-4eba-942a-b07a5a7afac7/ovn-acl-logging/0.log" Jul 07 16:16:03.048526 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:03.048480 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:16:22.815481 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:22.815443 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:16:23.323847 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:23.323816 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-10313d-openshift-default-697cf4bfbf-q5zcm"] Jul 07 16:16:23.324064 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:23.324038 2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-10313d-openshift-default-697cf4bfbf-q5zcm" podUID="c0d8c8f4-4a61-4842-bdb2-d9ad60f7c833" containerName="istio-proxy" containerID="cri-o://b8f281a419118677bbe348c7af6f6175431cda33b410c7441922b8aeecdf6cae" gracePeriod=30 Jul 07 16:16:23.330600 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:23.330576 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:16:23.668376 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:23.668291 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:16:23.857346 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:23.857317 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:16:24.273299 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:24.273256 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:16:24.450688 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:24.450653 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:16:24.553731 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:24.553633 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:16:24.871418 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:24.871386 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:16:25.986384 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:25.986340 2565 patch_prober.go:28] interesting pod/e2e-inf-a-10313d-openshift-default-697cf4bfbf-q5zcm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 07 16:16:25.986810 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:25.986404 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-10313d-openshift-default-697cf4bfbf-q5zcm" podUID="c0d8c8f4-4a61-4842-bdb2-d9ad60f7c833" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 07 16:16:28.588771 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:28.588712 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-10313d-openshift-default-697cf4bfbf-q5zcm" Jul 07 16:16:28.648098 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:28.648068 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0d8c8f4-4a61-4842-bdb2-d9ad60f7c833-workload-certs\") pod \"c0d8c8f4-4a61-4842-bdb2-d9ad60f7c833\" (UID: \"c0d8c8f4-4a61-4842-bdb2-d9ad60f7c833\") " Jul 07 16:16:28.648300 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:28.648116 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0d8c8f4-4a61-4842-bdb2-d9ad60f7c833-istiod-ca-cert\") pod \"c0d8c8f4-4a61-4842-bdb2-d9ad60f7c833\" (UID: \"c0d8c8f4-4a61-4842-bdb2-d9ad60f7c833\") " Jul 07 16:16:28.648300 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:28.648141 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0d8c8f4-4a61-4842-bdb2-d9ad60f7c833-workload-socket\") pod \"c0d8c8f4-4a61-4842-bdb2-d9ad60f7c833\" (UID: \"c0d8c8f4-4a61-4842-bdb2-d9ad60f7c833\") " Jul 07 16:16:28.648300 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:28.648262 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0d8c8f4-4a61-4842-bdb2-d9ad60f7c833-istio-token\") pod \"c0d8c8f4-4a61-4842-bdb2-d9ad60f7c833\" (UID: \"c0d8c8f4-4a61-4842-bdb2-d9ad60f7c833\") " Jul 07 16:16:28.648300 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:28.648300 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0d8c8f4-4a61-4842-bdb2-d9ad60f7c833-istio-podinfo\") pod \"c0d8c8f4-4a61-4842-bdb2-d9ad60f7c833\" (UID: \"c0d8c8f4-4a61-4842-bdb2-d9ad60f7c833\") " Jul 07 16:16:28.648534 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:28.648347 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0d8c8f4-4a61-4842-bdb2-d9ad60f7c833-credential-socket\") pod \"c0d8c8f4-4a61-4842-bdb2-d9ad60f7c833\" (UID: \"c0d8c8f4-4a61-4842-bdb2-d9ad60f7c833\") " Jul 07 16:16:28.648534 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:28.648423 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0d8c8f4-4a61-4842-bdb2-d9ad60f7c833-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "c0d8c8f4-4a61-4842-bdb2-d9ad60f7c833" (UID: "c0d8c8f4-4a61-4842-bdb2-d9ad60f7c833"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 16:16:28.648534 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:28.648471 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0d8c8f4-4a61-4842-bdb2-d9ad60f7c833-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "c0d8c8f4-4a61-4842-bdb2-d9ad60f7c833" (UID: "c0d8c8f4-4a61-4842-bdb2-d9ad60f7c833"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 16:16:28.648534 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:28.648515 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0d8c8f4-4a61-4842-bdb2-d9ad60f7c833-istio-data\") pod \"c0d8c8f4-4a61-4842-bdb2-d9ad60f7c833\" (UID: \"c0d8c8f4-4a61-4842-bdb2-d9ad60f7c833\") " Jul 07 16:16:28.648742 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:28.648575 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z8zwj\" (UniqueName: \"kubernetes.io/projected/c0d8c8f4-4a61-4842-bdb2-d9ad60f7c833-kube-api-access-z8zwj\") pod \"c0d8c8f4-4a61-4842-bdb2-d9ad60f7c833\" (UID: \"c0d8c8f4-4a61-4842-bdb2-d9ad60f7c833\") " Jul 07 16:16:28.648742 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:28.648617 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0d8c8f4-4a61-4842-bdb2-d9ad60f7c833-istio-envoy\") pod \"c0d8c8f4-4a61-4842-bdb2-d9ad60f7c833\" (UID: \"c0d8c8f4-4a61-4842-bdb2-d9ad60f7c833\") " Jul 07 16:16:28.648742 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:28.648571 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0d8c8f4-4a61-4842-bdb2-d9ad60f7c833-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "c0d8c8f4-4a61-4842-bdb2-d9ad60f7c833" (UID: "c0d8c8f4-4a61-4842-bdb2-d9ad60f7c833"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 16:16:28.648742 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:28.648659 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0d8c8f4-4a61-4842-bdb2-d9ad60f7c833-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "c0d8c8f4-4a61-4842-bdb2-d9ad60f7c833" (UID: "c0d8c8f4-4a61-4842-bdb2-d9ad60f7c833"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 16:16:28.648959 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:28.648935 2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0d8c8f4-4a61-4842-bdb2-d9ad60f7c833-workload-certs\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 07 16:16:28.649013 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:28.648961 2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0d8c8f4-4a61-4842-bdb2-d9ad60f7c833-istiod-ca-cert\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 07 16:16:28.649013 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:28.648974 2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0d8c8f4-4a61-4842-bdb2-d9ad60f7c833-workload-socket\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 07 16:16:28.649013 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:28.648986 2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0d8c8f4-4a61-4842-bdb2-d9ad60f7c833-credential-socket\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 07 16:16:28.649391 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:28.649370 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0d8c8f4-4a61-4842-bdb2-d9ad60f7c833-istio-data" (OuterVolumeSpecName: "istio-data") pod "c0d8c8f4-4a61-4842-bdb2-d9ad60f7c833" (UID: "c0d8c8f4-4a61-4842-bdb2-d9ad60f7c833"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 16:16:28.650573 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:28.650552 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0d8c8f4-4a61-4842-bdb2-d9ad60f7c833-kube-api-access-z8zwj" (OuterVolumeSpecName: "kube-api-access-z8zwj") pod "c0d8c8f4-4a61-4842-bdb2-d9ad60f7c833" (UID: "c0d8c8f4-4a61-4842-bdb2-d9ad60f7c833"). InnerVolumeSpecName "kube-api-access-z8zwj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 16:16:28.650928 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:28.650904 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0d8c8f4-4a61-4842-bdb2-d9ad60f7c833-istio-token" (OuterVolumeSpecName: "istio-token") pod "c0d8c8f4-4a61-4842-bdb2-d9ad60f7c833" (UID: "c0d8c8f4-4a61-4842-bdb2-d9ad60f7c833"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 16:16:28.651001 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:28.650932 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c0d8c8f4-4a61-4842-bdb2-d9ad60f7c833-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "c0d8c8f4-4a61-4842-bdb2-d9ad60f7c833" (UID: "c0d8c8f4-4a61-4842-bdb2-d9ad60f7c833"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 07 16:16:28.651073 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:28.651058 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0d8c8f4-4a61-4842-bdb2-d9ad60f7c833-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "c0d8c8f4-4a61-4842-bdb2-d9ad60f7c833" (UID: "c0d8c8f4-4a61-4842-bdb2-d9ad60f7c833"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 16:16:28.749920 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:28.749880 2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0d8c8f4-4a61-4842-bdb2-d9ad60f7c833-istio-token\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 07 16:16:28.749920 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:28.749909 2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0d8c8f4-4a61-4842-bdb2-d9ad60f7c833-istio-podinfo\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 07 16:16:28.749920 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:28.749918 2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0d8c8f4-4a61-4842-bdb2-d9ad60f7c833-istio-data\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 07 16:16:28.750150 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:28.749930 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z8zwj\" (UniqueName: \"kubernetes.io/projected/c0d8c8f4-4a61-4842-bdb2-d9ad60f7c833-kube-api-access-z8zwj\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 07 16:16:28.750150 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:28.749945 2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0d8c8f4-4a61-4842-bdb2-d9ad60f7c833-istio-envoy\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 07 16:16:28.811283 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:28.811251 2565 generic.go:356] "Generic (PLEG): container finished" podID="c0d8c8f4-4a61-4842-bdb2-d9ad60f7c833" containerID="b8f281a419118677bbe348c7af6f6175431cda33b410c7441922b8aeecdf6cae" exitCode=0 Jul 07 16:16:28.811468 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:28.811320 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-10313d-openshift-default-697cf4bfbf-q5zcm" Jul 07 16:16:28.811468 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:28.811341 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-10313d-openshift-default-697cf4bfbf-q5zcm" event={"ID":"c0d8c8f4-4a61-4842-bdb2-d9ad60f7c833","Type":"ContainerDied","Data":"b8f281a419118677bbe348c7af6f6175431cda33b410c7441922b8aeecdf6cae"} Jul 07 16:16:28.811468 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:28.811424 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-10313d-openshift-default-697cf4bfbf-q5zcm" event={"ID":"c0d8c8f4-4a61-4842-bdb2-d9ad60f7c833","Type":"ContainerDied","Data":"e3a160ac5fefce446c098888a8a444806231841148fb63fc81258e882f38f8ca"} Jul 07 16:16:28.811468 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:28.811449 2565 scope.go:117] "RemoveContainer" containerID="b8f281a419118677bbe348c7af6f6175431cda33b410c7441922b8aeecdf6cae" Jul 07 16:16:28.820859 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:28.820844 2565 scope.go:117] "RemoveContainer" containerID="0f27bf5e2618b699a1e42e79e429a79d1383fa5288753cee1d50659f6f85351e" Jul 07 16:16:28.828851 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:28.828834 2565 scope.go:117] "RemoveContainer" containerID="b8f281a419118677bbe348c7af6f6175431cda33b410c7441922b8aeecdf6cae" Jul 07 16:16:28.829103 ip-10-0-132-65 kubenswrapper[2565]: E0707 16:16:28.829084 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8f281a419118677bbe348c7af6f6175431cda33b410c7441922b8aeecdf6cae\": container with ID starting with b8f281a419118677bbe348c7af6f6175431cda33b410c7441922b8aeecdf6cae not found: ID does not exist" containerID="b8f281a419118677bbe348c7af6f6175431cda33b410c7441922b8aeecdf6cae" Jul 07 16:16:28.829149 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:28.829111 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8f281a419118677bbe348c7af6f6175431cda33b410c7441922b8aeecdf6cae"} err="failed to get container status \"b8f281a419118677bbe348c7af6f6175431cda33b410c7441922b8aeecdf6cae\": rpc error: code = NotFound desc = could not find container \"b8f281a419118677bbe348c7af6f6175431cda33b410c7441922b8aeecdf6cae\": container with ID starting with b8f281a419118677bbe348c7af6f6175431cda33b410c7441922b8aeecdf6cae not found: ID does not exist" Jul 07 16:16:28.829149 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:28.829128 2565 scope.go:117] "RemoveContainer" containerID="0f27bf5e2618b699a1e42e79e429a79d1383fa5288753cee1d50659f6f85351e" Jul 07 16:16:28.829362 ip-10-0-132-65 kubenswrapper[2565]: E0707 16:16:28.829345 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f27bf5e2618b699a1e42e79e429a79d1383fa5288753cee1d50659f6f85351e\": container with ID starting with 0f27bf5e2618b699a1e42e79e429a79d1383fa5288753cee1d50659f6f85351e not found: ID does not exist" containerID="0f27bf5e2618b699a1e42e79e429a79d1383fa5288753cee1d50659f6f85351e" Jul 07 16:16:28.829399 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:28.829368 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f27bf5e2618b699a1e42e79e429a79d1383fa5288753cee1d50659f6f85351e"} err="failed to get container status \"0f27bf5e2618b699a1e42e79e429a79d1383fa5288753cee1d50659f6f85351e\": rpc error: code = NotFound desc = could not find container \"0f27bf5e2618b699a1e42e79e429a79d1383fa5288753cee1d50659f6f85351e\": container with ID starting with 0f27bf5e2618b699a1e42e79e429a79d1383fa5288753cee1d50659f6f85351e not found: ID does not exist" Jul 07 16:16:28.839336 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:28.839281 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-10313d-openshift-default-697cf4bfbf-q5zcm"] Jul 07 16:16:28.847792 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:28.847773 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-10313d-openshift-default-697cf4bfbf-q5zcm"] Jul 07 16:16:29.663563 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:29.663530 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c0d8c8f4-4a61-4842-bdb2-d9ad60f7c833" path="/var/lib/kubelet/pods/c0d8c8f4-4a61-4842-bdb2-d9ad60f7c833/volumes" Jul 07 16:16:31.307357 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:31.307320 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:16:31.389881 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:31.389847 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:16:31.603980 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:31.603949 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-bda481-openshift-default-849c85bbf5-2jnt7"] Jul 07 16:16:31.604208 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:31.604173 2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-bda481-openshift-default-849c85bbf5-2jnt7" podUID="13c0936e-d449-4b9b-882e-587993a0df6f" containerName="istio-proxy" containerID="cri-o://2423ee310d70d2fc3dccdd53c697a553aeea8db977eec3e5c6756ff88cca77e8" gracePeriod=30 Jul 07 16:16:31.628069 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:31.628044 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:16:31.637287 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:31.637257 2565 patch_prober.go:28] interesting pod/e2e-inf-b-bda481-openshift-default-849c85bbf5-2jnt7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 07 16:16:31.637389 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:31.637320 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-bda481-openshift-default-849c85bbf5-2jnt7" podUID="13c0936e-d449-4b9b-882e-587993a0df6f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 07 16:16:31.845727 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:31.845692 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:16:32.213843 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:32.213809 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:16:32.544858 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:32.544780 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-78b5af-openshift-default-75c949ffb9-7rvw8"] Jul 07 16:16:32.545240 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:32.545163 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c0d8c8f4-4a61-4842-bdb2-d9ad60f7c833" containerName="istio-proxy" Jul 07 16:16:32.545240 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:32.545175 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="c0d8c8f4-4a61-4842-bdb2-d9ad60f7c833" containerName="istio-proxy" Jul 07 16:16:32.545240 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:32.545185 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dca7b67e-1b6f-4572-861c-4a22277d5100" containerName="maas-api" Jul 07 16:16:32.545240 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:32.545191 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="dca7b67e-1b6f-4572-861c-4a22277d5100" containerName="maas-api" Jul 07 16:16:32.545240 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:32.545202 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c0d8c8f4-4a61-4842-bdb2-d9ad60f7c833" containerName="istio-proxy" Jul 07 16:16:32.545240 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:32.545208 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="c0d8c8f4-4a61-4842-bdb2-d9ad60f7c833" containerName="istio-proxy" Jul 07 16:16:32.545443 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:32.545263 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="c0d8c8f4-4a61-4842-bdb2-d9ad60f7c833" containerName="istio-proxy" Jul 07 16:16:32.545443 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:32.545273 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="c0d8c8f4-4a61-4842-bdb2-d9ad60f7c833" containerName="istio-proxy" Jul 07 16:16:32.545443 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:32.545280 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="dca7b67e-1b6f-4572-861c-4a22277d5100" containerName="maas-api" Jul 07 16:16:32.549468 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:32.549447 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-78b5af-openshift-default-75c949ffb9-7rvw8" Jul 07 16:16:32.552979 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:32.552955 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-78b5af-openshift-default-dockercfg-l9p66\"" Jul 07 16:16:32.568478 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:32.568452 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-78b5af-openshift-default-75c949ffb9-7rvw8"] Jul 07 16:16:32.579666 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:32.579625 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:16:32.685244 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:32.685210 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wfjlc\" (UniqueName: \"kubernetes.io/projected/8fe453f9-345b-4288-8e90-ee666c180de9-kube-api-access-wfjlc\") pod \"e2e-api-a-78b5af-openshift-default-75c949ffb9-7rvw8\" (UID: \"8fe453f9-345b-4288-8e90-ee666c180de9\") " pod="openshift-ingress/e2e-api-a-78b5af-openshift-default-75c949ffb9-7rvw8" Jul 07 16:16:32.685244 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:32.685253 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8fe453f9-345b-4288-8e90-ee666c180de9-istio-podinfo\") pod \"e2e-api-a-78b5af-openshift-default-75c949ffb9-7rvw8\" (UID: \"8fe453f9-345b-4288-8e90-ee666c180de9\") " pod="openshift-ingress/e2e-api-a-78b5af-openshift-default-75c949ffb9-7rvw8" Jul 07 16:16:32.685552 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:32.685278 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8fe453f9-345b-4288-8e90-ee666c180de9-istio-data\") pod \"e2e-api-a-78b5af-openshift-default-75c949ffb9-7rvw8\" (UID: \"8fe453f9-345b-4288-8e90-ee666c180de9\") " pod="openshift-ingress/e2e-api-a-78b5af-openshift-default-75c949ffb9-7rvw8" Jul 07 16:16:32.685552 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:32.685361 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8fe453f9-345b-4288-8e90-ee666c180de9-istiod-ca-cert\") pod \"e2e-api-a-78b5af-openshift-default-75c949ffb9-7rvw8\" (UID: \"8fe453f9-345b-4288-8e90-ee666c180de9\") " pod="openshift-ingress/e2e-api-a-78b5af-openshift-default-75c949ffb9-7rvw8" Jul 07 16:16:32.685552 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:32.685398 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fe453f9-345b-4288-8e90-ee666c180de9-workload-socket\") pod \"e2e-api-a-78b5af-openshift-default-75c949ffb9-7rvw8\" (UID: \"8fe453f9-345b-4288-8e90-ee666c180de9\") " pod="openshift-ingress/e2e-api-a-78b5af-openshift-default-75c949ffb9-7rvw8" Jul 07 16:16:32.685552 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:32.685415 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8fe453f9-345b-4288-8e90-ee666c180de9-istio-token\") pod \"e2e-api-a-78b5af-openshift-default-75c949ffb9-7rvw8\" (UID: \"8fe453f9-345b-4288-8e90-ee666c180de9\") " pod="openshift-ingress/e2e-api-a-78b5af-openshift-default-75c949ffb9-7rvw8" Jul 07 16:16:32.685552 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:32.685517 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8fe453f9-345b-4288-8e90-ee666c180de9-workload-certs\") pod \"e2e-api-a-78b5af-openshift-default-75c949ffb9-7rvw8\" (UID: \"8fe453f9-345b-4288-8e90-ee666c180de9\") " pod="openshift-ingress/e2e-api-a-78b5af-openshift-default-75c949ffb9-7rvw8" Jul 07 16:16:32.685808 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:32.685595 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8fe453f9-345b-4288-8e90-ee666c180de9-istio-envoy\") pod \"e2e-api-a-78b5af-openshift-default-75c949ffb9-7rvw8\" (UID: \"8fe453f9-345b-4288-8e90-ee666c180de9\") " pod="openshift-ingress/e2e-api-a-78b5af-openshift-default-75c949ffb9-7rvw8" Jul 07 16:16:32.685808 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:32.685637 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fe453f9-345b-4288-8e90-ee666c180de9-credential-socket\") pod \"e2e-api-a-78b5af-openshift-default-75c949ffb9-7rvw8\" (UID: \"8fe453f9-345b-4288-8e90-ee666c180de9\") " pod="openshift-ingress/e2e-api-a-78b5af-openshift-default-75c949ffb9-7rvw8" Jul 07 16:16:32.773232 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:32.773194 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:16:32.786382 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:32.786352 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fe453f9-345b-4288-8e90-ee666c180de9-credential-socket\") pod \"e2e-api-a-78b5af-openshift-default-75c949ffb9-7rvw8\" (UID: \"8fe453f9-345b-4288-8e90-ee666c180de9\") " pod="openshift-ingress/e2e-api-a-78b5af-openshift-default-75c949ffb9-7rvw8" Jul 07 16:16:32.786529 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:32.786403 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wfjlc\" (UniqueName: \"kubernetes.io/projected/8fe453f9-345b-4288-8e90-ee666c180de9-kube-api-access-wfjlc\") pod \"e2e-api-a-78b5af-openshift-default-75c949ffb9-7rvw8\" (UID: \"8fe453f9-345b-4288-8e90-ee666c180de9\") " pod="openshift-ingress/e2e-api-a-78b5af-openshift-default-75c949ffb9-7rvw8" Jul 07 16:16:32.786529 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:32.786425 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8fe453f9-345b-4288-8e90-ee666c180de9-istio-podinfo\") pod \"e2e-api-a-78b5af-openshift-default-75c949ffb9-7rvw8\" (UID: \"8fe453f9-345b-4288-8e90-ee666c180de9\") " pod="openshift-ingress/e2e-api-a-78b5af-openshift-default-75c949ffb9-7rvw8" Jul 07 16:16:32.786529 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:32.786449 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8fe453f9-345b-4288-8e90-ee666c180de9-istio-data\") pod \"e2e-api-a-78b5af-openshift-default-75c949ffb9-7rvw8\" (UID: \"8fe453f9-345b-4288-8e90-ee666c180de9\") " pod="openshift-ingress/e2e-api-a-78b5af-openshift-default-75c949ffb9-7rvw8" Jul 07 16:16:32.786529 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:32.786470 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8fe453f9-345b-4288-8e90-ee666c180de9-istiod-ca-cert\") pod \"e2e-api-a-78b5af-openshift-default-75c949ffb9-7rvw8\" (UID: \"8fe453f9-345b-4288-8e90-ee666c180de9\") " pod="openshift-ingress/e2e-api-a-78b5af-openshift-default-75c949ffb9-7rvw8" Jul 07 16:16:32.786529 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:32.786522 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fe453f9-345b-4288-8e90-ee666c180de9-workload-socket\") pod \"e2e-api-a-78b5af-openshift-default-75c949ffb9-7rvw8\" (UID: \"8fe453f9-345b-4288-8e90-ee666c180de9\") " pod="openshift-ingress/e2e-api-a-78b5af-openshift-default-75c949ffb9-7rvw8" Jul 07 16:16:32.786802 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:32.786545 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8fe453f9-345b-4288-8e90-ee666c180de9-istio-token\") pod \"e2e-api-a-78b5af-openshift-default-75c949ffb9-7rvw8\" (UID: \"8fe453f9-345b-4288-8e90-ee666c180de9\") " pod="openshift-ingress/e2e-api-a-78b5af-openshift-default-75c949ffb9-7rvw8" Jul 07 16:16:32.786802 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:32.786586 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8fe453f9-345b-4288-8e90-ee666c180de9-workload-certs\") pod \"e2e-api-a-78b5af-openshift-default-75c949ffb9-7rvw8\" (UID: \"8fe453f9-345b-4288-8e90-ee666c180de9\") " pod="openshift-ingress/e2e-api-a-78b5af-openshift-default-75c949ffb9-7rvw8" Jul 07 16:16:32.786802 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:32.786672 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8fe453f9-345b-4288-8e90-ee666c180de9-istio-envoy\") pod \"e2e-api-a-78b5af-openshift-default-75c949ffb9-7rvw8\" (UID: \"8fe453f9-345b-4288-8e90-ee666c180de9\") " pod="openshift-ingress/e2e-api-a-78b5af-openshift-default-75c949ffb9-7rvw8" Jul 07 16:16:32.786959 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:32.786820 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fe453f9-345b-4288-8e90-ee666c180de9-credential-socket\") pod \"e2e-api-a-78b5af-openshift-default-75c949ffb9-7rvw8\" (UID: \"8fe453f9-345b-4288-8e90-ee666c180de9\") " pod="openshift-ingress/e2e-api-a-78b5af-openshift-default-75c949ffb9-7rvw8" Jul 07 16:16:32.786959 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:32.786876 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8fe453f9-345b-4288-8e90-ee666c180de9-istio-data\") pod \"e2e-api-a-78b5af-openshift-default-75c949ffb9-7rvw8\" (UID: \"8fe453f9-345b-4288-8e90-ee666c180de9\") " pod="openshift-ingress/e2e-api-a-78b5af-openshift-default-75c949ffb9-7rvw8" Jul 07 16:16:32.787047 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:32.787018 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fe453f9-345b-4288-8e90-ee666c180de9-workload-socket\") pod \"e2e-api-a-78b5af-openshift-default-75c949ffb9-7rvw8\" (UID: \"8fe453f9-345b-4288-8e90-ee666c180de9\") " pod="openshift-ingress/e2e-api-a-78b5af-openshift-default-75c949ffb9-7rvw8" Jul 07 16:16:32.787192 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:32.787172 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8fe453f9-345b-4288-8e90-ee666c180de9-workload-certs\") pod \"e2e-api-a-78b5af-openshift-default-75c949ffb9-7rvw8\" (UID: \"8fe453f9-345b-4288-8e90-ee666c180de9\") " pod="openshift-ingress/e2e-api-a-78b5af-openshift-default-75c949ffb9-7rvw8" Jul 07 16:16:32.787372 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:32.787347 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8fe453f9-345b-4288-8e90-ee666c180de9-istiod-ca-cert\") pod \"e2e-api-a-78b5af-openshift-default-75c949ffb9-7rvw8\" (UID: \"8fe453f9-345b-4288-8e90-ee666c180de9\") " pod="openshift-ingress/e2e-api-a-78b5af-openshift-default-75c949ffb9-7rvw8" Jul 07 16:16:32.788863 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:32.788844 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8fe453f9-345b-4288-8e90-ee666c180de9-istio-envoy\") pod \"e2e-api-a-78b5af-openshift-default-75c949ffb9-7rvw8\" (UID: \"8fe453f9-345b-4288-8e90-ee666c180de9\") " pod="openshift-ingress/e2e-api-a-78b5af-openshift-default-75c949ffb9-7rvw8" Jul 07 16:16:32.789116 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:32.789096 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8fe453f9-345b-4288-8e90-ee666c180de9-istio-podinfo\") pod \"e2e-api-a-78b5af-openshift-default-75c949ffb9-7rvw8\" (UID: \"8fe453f9-345b-4288-8e90-ee666c180de9\") " pod="openshift-ingress/e2e-api-a-78b5af-openshift-default-75c949ffb9-7rvw8" Jul 07 16:16:32.796600 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:32.796543 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8fe453f9-345b-4288-8e90-ee666c180de9-istio-token\") pod \"e2e-api-a-78b5af-openshift-default-75c949ffb9-7rvw8\" (UID: \"8fe453f9-345b-4288-8e90-ee666c180de9\") " pod="openshift-ingress/e2e-api-a-78b5af-openshift-default-75c949ffb9-7rvw8" Jul 07 16:16:32.796713 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:32.796695 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wfjlc\" (UniqueName: \"kubernetes.io/projected/8fe453f9-345b-4288-8e90-ee666c180de9-kube-api-access-wfjlc\") pod \"e2e-api-a-78b5af-openshift-default-75c949ffb9-7rvw8\" (UID: \"8fe453f9-345b-4288-8e90-ee666c180de9\") " pod="openshift-ingress/e2e-api-a-78b5af-openshift-default-75c949ffb9-7rvw8" Jul 07 16:16:32.860184 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:32.860149 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-78b5af-openshift-default-75c949ffb9-7rvw8" Jul 07 16:16:33.013649 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:33.013626 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-78b5af-openshift-default-75c949ffb9-7rvw8"] Jul 07 16:16:33.015383 ip-10-0-132-65 kubenswrapper[2565]: W0707 16:16:33.015357 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fe453f9_345b_4288_8e90_ee666c180de9.slice/crio-98638597a0383c6395212d93ffc072379f292c9152112edf9f24a8c4ba8ee0a0 WatchSource:0}: Error finding container 98638597a0383c6395212d93ffc072379f292c9152112edf9f24a8c4ba8ee0a0: Status 404 returned error can't find the container with id 98638597a0383c6395212d93ffc072379f292c9152112edf9f24a8c4ba8ee0a0 Jul 07 16:16:33.017517 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:33.017463 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 07 16:16:33.017617 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:33.017566 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 07 16:16:33.017665 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:33.017627 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 07 16:16:33.839169 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:33.839125 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-78b5af-openshift-default-75c949ffb9-7rvw8" event={"ID":"8fe453f9-345b-4288-8e90-ee666c180de9","Type":"ContainerStarted","Data":"f008c76e5c537e94b95c42933ff2976ff640855e8d455b444ff28121b5f538b3"} Jul 07 16:16:33.839592 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:33.839187 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-78b5af-openshift-default-75c949ffb9-7rvw8" event={"ID":"8fe453f9-345b-4288-8e90-ee666c180de9","Type":"ContainerStarted","Data":"98638597a0383c6395212d93ffc072379f292c9152112edf9f24a8c4ba8ee0a0"} Jul 07 16:16:33.860466 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:33.860433 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-78b5af-openshift-default-75c949ffb9-7rvw8" Jul 07 16:16:33.868975 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:33.868925 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-78b5af-openshift-default-75c949ffb9-7rvw8" podStartSLOduration=1.868902497 podStartE2EDuration="1.868902497s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:16:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:16:33.867887996 +0000 UTC m=+2132.824465465" watchObservedRunningTime="2026-07-07 16:16:33.868902497 +0000 UTC m=+2132.825479965" Jul 07 16:16:34.866312 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:34.866280 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-78b5af-openshift-default-75c949ffb9-7rvw8" Jul 07 16:16:35.840622 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:35.840577 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:16:35.850462 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:35.850432 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-78b5af-openshift-default-75c949ffb9-7rvw8" Jul 07 16:16:35.851438 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:35.851418 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-78b5af-openshift-default-75c949ffb9-7rvw8" Jul 07 16:16:35.941553 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:35.941523 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:16:36.089401 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:36.089364 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:16:36.289355 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:36.289266 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:16:36.610262 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:36.610215 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:16:36.705937 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:36.705907 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:16:36.857358 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:36.857331 2565 generic.go:356] "Generic (PLEG): container finished" podID="13c0936e-d449-4b9b-882e-587993a0df6f" containerID="2423ee310d70d2fc3dccdd53c697a553aeea8db977eec3e5c6756ff88cca77e8" exitCode=0 Jul 07 16:16:36.857546 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:36.857399 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-bda481-openshift-default-849c85bbf5-2jnt7" event={"ID":"13c0936e-d449-4b9b-882e-587993a0df6f","Type":"ContainerDied","Data":"2423ee310d70d2fc3dccdd53c697a553aeea8db977eec3e5c6756ff88cca77e8"} Jul 07 16:16:36.857546 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:36.857459 2565 scope.go:117] "RemoveContainer" containerID="b5954bc8b67181ea56e930a6330aa2c88c7ab359c69efdfb25a4ac6915c37309" Jul 07 16:16:36.873134 ip-10-0-132-65 kubenswrapper[2565]: E0707 16:16:36.873112 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5954bc8b67181ea56e930a6330aa2c88c7ab359c69efdfb25a4ac6915c37309\": container with ID starting with b5954bc8b67181ea56e930a6330aa2c88c7ab359c69efdfb25a4ac6915c37309 not found: ID does not exist" containerID="b5954bc8b67181ea56e930a6330aa2c88c7ab359c69efdfb25a4ac6915c37309" Jul 07 16:16:36.876581 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:36.876565 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-bda481-openshift-default-849c85bbf5-2jnt7" Jul 07 16:16:37.036860 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:37.036827 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/13c0936e-d449-4b9b-882e-587993a0df6f-credential-socket\") pod \"13c0936e-d449-4b9b-882e-587993a0df6f\" (UID: \"13c0936e-d449-4b9b-882e-587993a0df6f\") " Jul 07 16:16:37.037336 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:37.036887 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/13c0936e-d449-4b9b-882e-587993a0df6f-istiod-ca-cert\") pod \"13c0936e-d449-4b9b-882e-587993a0df6f\" (UID: \"13c0936e-d449-4b9b-882e-587993a0df6f\") " Jul 07 16:16:37.037336 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:37.036909 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/13c0936e-d449-4b9b-882e-587993a0df6f-istio-podinfo\") pod \"13c0936e-d449-4b9b-882e-587993a0df6f\" (UID: \"13c0936e-d449-4b9b-882e-587993a0df6f\") " Jul 07 16:16:37.037336 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:37.036928 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xrknk\" (UniqueName: \"kubernetes.io/projected/13c0936e-d449-4b9b-882e-587993a0df6f-kube-api-access-xrknk\") pod \"13c0936e-d449-4b9b-882e-587993a0df6f\" (UID: \"13c0936e-d449-4b9b-882e-587993a0df6f\") " Jul 07 16:16:37.037336 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:37.036964 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/13c0936e-d449-4b9b-882e-587993a0df6f-istio-token\") pod \"13c0936e-d449-4b9b-882e-587993a0df6f\" (UID: \"13c0936e-d449-4b9b-882e-587993a0df6f\") " Jul 07 16:16:37.037336 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:37.036987 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/13c0936e-d449-4b9b-882e-587993a0df6f-workload-socket\") pod \"13c0936e-d449-4b9b-882e-587993a0df6f\" (UID: \"13c0936e-d449-4b9b-882e-587993a0df6f\") " Jul 07 16:16:37.037336 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:37.037027 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/13c0936e-d449-4b9b-882e-587993a0df6f-istio-envoy\") pod \"13c0936e-d449-4b9b-882e-587993a0df6f\" (UID: \"13c0936e-d449-4b9b-882e-587993a0df6f\") " Jul 07 16:16:37.037336 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:37.037084 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/13c0936e-d449-4b9b-882e-587993a0df6f-workload-certs\") pod \"13c0936e-d449-4b9b-882e-587993a0df6f\" (UID: \"13c0936e-d449-4b9b-882e-587993a0df6f\") " Jul 07 16:16:37.037336 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:37.037124 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/13c0936e-d449-4b9b-882e-587993a0df6f-istio-data\") pod \"13c0936e-d449-4b9b-882e-587993a0df6f\" (UID: \"13c0936e-d449-4b9b-882e-587993a0df6f\") " Jul 07 16:16:37.037336 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:37.037142 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13c0936e-d449-4b9b-882e-587993a0df6f-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "13c0936e-d449-4b9b-882e-587993a0df6f" (UID: "13c0936e-d449-4b9b-882e-587993a0df6f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 16:16:37.037336 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:37.037255 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13c0936e-d449-4b9b-882e-587993a0df6f-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "13c0936e-d449-4b9b-882e-587993a0df6f" (UID: "13c0936e-d449-4b9b-882e-587993a0df6f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 16:16:37.037903 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:37.037381 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13c0936e-d449-4b9b-882e-587993a0df6f-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "13c0936e-d449-4b9b-882e-587993a0df6f" (UID: "13c0936e-d449-4b9b-882e-587993a0df6f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 16:16:37.037903 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:37.037557 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13c0936e-d449-4b9b-882e-587993a0df6f-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "13c0936e-d449-4b9b-882e-587993a0df6f" (UID: "13c0936e-d449-4b9b-882e-587993a0df6f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 16:16:37.037903 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:37.037574 2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/13c0936e-d449-4b9b-882e-587993a0df6f-credential-socket\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 07 16:16:37.037903 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:37.037594 2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/13c0936e-d449-4b9b-882e-587993a0df6f-istiod-ca-cert\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 07 16:16:37.037903 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:37.037606 2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/13c0936e-d449-4b9b-882e-587993a0df6f-workload-socket\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 07 16:16:37.038275 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:37.038250 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13c0936e-d449-4b9b-882e-587993a0df6f-istio-data" (OuterVolumeSpecName: "istio-data") pod "13c0936e-d449-4b9b-882e-587993a0df6f" (UID: "13c0936e-d449-4b9b-882e-587993a0df6f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 16:16:37.039300 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:37.039274 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/13c0936e-d449-4b9b-882e-587993a0df6f-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "13c0936e-d449-4b9b-882e-587993a0df6f" (UID: "13c0936e-d449-4b9b-882e-587993a0df6f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 07 16:16:37.039414 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:37.039383 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13c0936e-d449-4b9b-882e-587993a0df6f-kube-api-access-xrknk" (OuterVolumeSpecName: "kube-api-access-xrknk") pod "13c0936e-d449-4b9b-882e-587993a0df6f" (UID: "13c0936e-d449-4b9b-882e-587993a0df6f"). InnerVolumeSpecName "kube-api-access-xrknk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 16:16:37.039505 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:37.039473 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13c0936e-d449-4b9b-882e-587993a0df6f-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "13c0936e-d449-4b9b-882e-587993a0df6f" (UID: "13c0936e-d449-4b9b-882e-587993a0df6f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 16:16:37.039570 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:37.039551 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13c0936e-d449-4b9b-882e-587993a0df6f-istio-token" (OuterVolumeSpecName: "istio-token") pod "13c0936e-d449-4b9b-882e-587993a0df6f" (UID: "13c0936e-d449-4b9b-882e-587993a0df6f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 16:16:37.138589 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:37.138482 2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/13c0936e-d449-4b9b-882e-587993a0df6f-istio-token\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 07 16:16:37.138589 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:37.138547 2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/13c0936e-d449-4b9b-882e-587993a0df6f-istio-envoy\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 07 16:16:37.138589 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:37.138556 2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/13c0936e-d449-4b9b-882e-587993a0df6f-workload-certs\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 07 16:16:37.138589 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:37.138569 2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/13c0936e-d449-4b9b-882e-587993a0df6f-istio-data\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 07 16:16:37.138589 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:37.138577 2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/13c0936e-d449-4b9b-882e-587993a0df6f-istio-podinfo\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 07 16:16:37.138589 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:37.138585 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xrknk\" (UniqueName: \"kubernetes.io/projected/13c0936e-d449-4b9b-882e-587993a0df6f-kube-api-access-xrknk\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 07 16:16:37.864034 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:37.864000 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-bda481-openshift-default-849c85bbf5-2jnt7" Jul 07 16:16:37.864231 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:37.864039 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-bda481-openshift-default-849c85bbf5-2jnt7" event={"ID":"13c0936e-d449-4b9b-882e-587993a0df6f","Type":"ContainerDied","Data":"b59afbfd78d58516193b53d5f60de198ce75f7ab06d9e0d0747ba652b6b8a0dd"} Jul 07 16:16:37.864231 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:37.864106 2565 scope.go:117] "RemoveContainer" containerID="2423ee310d70d2fc3dccdd53c697a553aeea8db977eec3e5c6756ff88cca77e8" Jul 07 16:16:37.903250 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:37.903223 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-bda481-openshift-default-849c85bbf5-2jnt7"] Jul 07 16:16:37.915728 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:37.915702 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-bda481-openshift-default-849c85bbf5-2jnt7"] Jul 07 16:16:39.661255 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:39.661224 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="13c0936e-d449-4b9b-882e-587993a0df6f" path="/var/lib/kubelet/pods/13c0936e-d449-4b9b-882e-587993a0df6f/volumes" Jul 07 16:16:40.401615 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:40.401578 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:16:40.882465 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:40.882428 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:16:41.184777 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:41.184693 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-b728d9-openshift-default-6f85f8ff65-gnstx"] Jul 07 16:16:41.185177 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:41.185158 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="13c0936e-d449-4b9b-882e-587993a0df6f" containerName="istio-proxy" Jul 07 16:16:41.185269 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:41.185179 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="13c0936e-d449-4b9b-882e-587993a0df6f" containerName="istio-proxy" Jul 07 16:16:41.185326 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:41.185302 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="13c0936e-d449-4b9b-882e-587993a0df6f" containerName="istio-proxy" Jul 07 16:16:41.185326 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:41.185316 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="13c0936e-d449-4b9b-882e-587993a0df6f" containerName="istio-proxy" Jul 07 16:16:41.189906 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:41.189885 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-b728d9-openshift-default-6f85f8ff65-gnstx" Jul 07 16:16:41.193882 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:41.193770 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-b728d9-openshift-default-dockercfg-t5qnk\"" Jul 07 16:16:41.203287 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:41.203260 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-b728d9-openshift-default-6f85f8ff65-gnstx"] Jul 07 16:16:41.276563 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:41.276526 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0aae1a92-e503-48d7-8aa3-a04792622460-istio-data\") pod \"e2e-api-b-b728d9-openshift-default-6f85f8ff65-gnstx\" (UID: \"0aae1a92-e503-48d7-8aa3-a04792622460\") " pod="openshift-ingress/e2e-api-b-b728d9-openshift-default-6f85f8ff65-gnstx" Jul 07 16:16:41.276767 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:41.276582 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0aae1a92-e503-48d7-8aa3-a04792622460-workload-socket\") pod \"e2e-api-b-b728d9-openshift-default-6f85f8ff65-gnstx\" (UID: \"0aae1a92-e503-48d7-8aa3-a04792622460\") " pod="openshift-ingress/e2e-api-b-b728d9-openshift-default-6f85f8ff65-gnstx" Jul 07 16:16:41.276767 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:41.276629 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0aae1a92-e503-48d7-8aa3-a04792622460-istio-token\") pod \"e2e-api-b-b728d9-openshift-default-6f85f8ff65-gnstx\" (UID: \"0aae1a92-e503-48d7-8aa3-a04792622460\") " pod="openshift-ingress/e2e-api-b-b728d9-openshift-default-6f85f8ff65-gnstx" Jul 07 16:16:41.276767 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:41.276672 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0aae1a92-e503-48d7-8aa3-a04792622460-istio-podinfo\") pod \"e2e-api-b-b728d9-openshift-default-6f85f8ff65-gnstx\" (UID: \"0aae1a92-e503-48d7-8aa3-a04792622460\") " pod="openshift-ingress/e2e-api-b-b728d9-openshift-default-6f85f8ff65-gnstx" Jul 07 16:16:41.276767 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:41.276688 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rl878\" (UniqueName: \"kubernetes.io/projected/0aae1a92-e503-48d7-8aa3-a04792622460-kube-api-access-rl878\") pod \"e2e-api-b-b728d9-openshift-default-6f85f8ff65-gnstx\" (UID: \"0aae1a92-e503-48d7-8aa3-a04792622460\") " pod="openshift-ingress/e2e-api-b-b728d9-openshift-default-6f85f8ff65-gnstx" Jul 07 16:16:41.276767 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:41.276707 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0aae1a92-e503-48d7-8aa3-a04792622460-istio-envoy\") pod \"e2e-api-b-b728d9-openshift-default-6f85f8ff65-gnstx\" (UID: \"0aae1a92-e503-48d7-8aa3-a04792622460\") " pod="openshift-ingress/e2e-api-b-b728d9-openshift-default-6f85f8ff65-gnstx" Jul 07 16:16:41.277018 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:41.276804 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0aae1a92-e503-48d7-8aa3-a04792622460-credential-socket\") pod \"e2e-api-b-b728d9-openshift-default-6f85f8ff65-gnstx\" (UID: \"0aae1a92-e503-48d7-8aa3-a04792622460\") " pod="openshift-ingress/e2e-api-b-b728d9-openshift-default-6f85f8ff65-gnstx" Jul 07 16:16:41.277018 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:41.276868 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0aae1a92-e503-48d7-8aa3-a04792622460-istiod-ca-cert\") pod \"e2e-api-b-b728d9-openshift-default-6f85f8ff65-gnstx\" (UID: \"0aae1a92-e503-48d7-8aa3-a04792622460\") " pod="openshift-ingress/e2e-api-b-b728d9-openshift-default-6f85f8ff65-gnstx" Jul 07 16:16:41.277018 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:41.276907 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0aae1a92-e503-48d7-8aa3-a04792622460-workload-certs\") pod \"e2e-api-b-b728d9-openshift-default-6f85f8ff65-gnstx\" (UID: \"0aae1a92-e503-48d7-8aa3-a04792622460\") " pod="openshift-ingress/e2e-api-b-b728d9-openshift-default-6f85f8ff65-gnstx" Jul 07 16:16:41.280582 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:41.280559 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:16:41.377679 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:41.377643 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0aae1a92-e503-48d7-8aa3-a04792622460-credential-socket\") pod \"e2e-api-b-b728d9-openshift-default-6f85f8ff65-gnstx\" (UID: \"0aae1a92-e503-48d7-8aa3-a04792622460\") " pod="openshift-ingress/e2e-api-b-b728d9-openshift-default-6f85f8ff65-gnstx" Jul 07 16:16:41.377861 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:41.377700 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0aae1a92-e503-48d7-8aa3-a04792622460-istiod-ca-cert\") pod \"e2e-api-b-b728d9-openshift-default-6f85f8ff65-gnstx\" (UID: \"0aae1a92-e503-48d7-8aa3-a04792622460\") " pod="openshift-ingress/e2e-api-b-b728d9-openshift-default-6f85f8ff65-gnstx" Jul 07 16:16:41.377861 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:41.377733 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0aae1a92-e503-48d7-8aa3-a04792622460-workload-certs\") pod \"e2e-api-b-b728d9-openshift-default-6f85f8ff65-gnstx\" (UID: \"0aae1a92-e503-48d7-8aa3-a04792622460\") " pod="openshift-ingress/e2e-api-b-b728d9-openshift-default-6f85f8ff65-gnstx" Jul 07 16:16:41.377861 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:41.377770 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0aae1a92-e503-48d7-8aa3-a04792622460-istio-data\") pod \"e2e-api-b-b728d9-openshift-default-6f85f8ff65-gnstx\" (UID: \"0aae1a92-e503-48d7-8aa3-a04792622460\") " pod="openshift-ingress/e2e-api-b-b728d9-openshift-default-6f85f8ff65-gnstx" Jul 07 16:16:41.377861 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:41.377814 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0aae1a92-e503-48d7-8aa3-a04792622460-workload-socket\") pod \"e2e-api-b-b728d9-openshift-default-6f85f8ff65-gnstx\" (UID: \"0aae1a92-e503-48d7-8aa3-a04792622460\") " pod="openshift-ingress/e2e-api-b-b728d9-openshift-default-6f85f8ff65-gnstx" Jul 07 16:16:41.378067 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:41.377861 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0aae1a92-e503-48d7-8aa3-a04792622460-istio-token\") pod \"e2e-api-b-b728d9-openshift-default-6f85f8ff65-gnstx\" (UID: \"0aae1a92-e503-48d7-8aa3-a04792622460\") " pod="openshift-ingress/e2e-api-b-b728d9-openshift-default-6f85f8ff65-gnstx" Jul 07 16:16:41.378067 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:41.377905 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0aae1a92-e503-48d7-8aa3-a04792622460-istio-podinfo\") pod \"e2e-api-b-b728d9-openshift-default-6f85f8ff65-gnstx\" (UID: \"0aae1a92-e503-48d7-8aa3-a04792622460\") " pod="openshift-ingress/e2e-api-b-b728d9-openshift-default-6f85f8ff65-gnstx" Jul 07 16:16:41.378067 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:41.377931 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rl878\" (UniqueName: \"kubernetes.io/projected/0aae1a92-e503-48d7-8aa3-a04792622460-kube-api-access-rl878\") pod \"e2e-api-b-b728d9-openshift-default-6f85f8ff65-gnstx\" (UID: \"0aae1a92-e503-48d7-8aa3-a04792622460\") " pod="openshift-ingress/e2e-api-b-b728d9-openshift-default-6f85f8ff65-gnstx" Jul 07 16:16:41.378205 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:41.378099 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0aae1a92-e503-48d7-8aa3-a04792622460-credential-socket\") pod \"e2e-api-b-b728d9-openshift-default-6f85f8ff65-gnstx\" (UID: \"0aae1a92-e503-48d7-8aa3-a04792622460\") " pod="openshift-ingress/e2e-api-b-b728d9-openshift-default-6f85f8ff65-gnstx" Jul 07 16:16:41.378205 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:41.378127 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0aae1a92-e503-48d7-8aa3-a04792622460-istio-envoy\") pod \"e2e-api-b-b728d9-openshift-default-6f85f8ff65-gnstx\" (UID: \"0aae1a92-e503-48d7-8aa3-a04792622460\") " pod="openshift-ingress/e2e-api-b-b728d9-openshift-default-6f85f8ff65-gnstx" Jul 07 16:16:41.378309 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:41.378290 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0aae1a92-e503-48d7-8aa3-a04792622460-workload-certs\") pod \"e2e-api-b-b728d9-openshift-default-6f85f8ff65-gnstx\" (UID: \"0aae1a92-e503-48d7-8aa3-a04792622460\") " pod="openshift-ingress/e2e-api-b-b728d9-openshift-default-6f85f8ff65-gnstx" Jul 07 16:16:41.378471 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:41.378451 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0aae1a92-e503-48d7-8aa3-a04792622460-istio-data\") pod \"e2e-api-b-b728d9-openshift-default-6f85f8ff65-gnstx\" (UID: \"0aae1a92-e503-48d7-8aa3-a04792622460\") " pod="openshift-ingress/e2e-api-b-b728d9-openshift-default-6f85f8ff65-gnstx" Jul 07 16:16:41.378576 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:41.378533 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0aae1a92-e503-48d7-8aa3-a04792622460-istiod-ca-cert\") pod \"e2e-api-b-b728d9-openshift-default-6f85f8ff65-gnstx\" (UID: \"0aae1a92-e503-48d7-8aa3-a04792622460\") " pod="openshift-ingress/e2e-api-b-b728d9-openshift-default-6f85f8ff65-gnstx" Jul 07 16:16:41.378576 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:41.378539 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0aae1a92-e503-48d7-8aa3-a04792622460-workload-socket\") pod \"e2e-api-b-b728d9-openshift-default-6f85f8ff65-gnstx\" (UID: \"0aae1a92-e503-48d7-8aa3-a04792622460\") " pod="openshift-ingress/e2e-api-b-b728d9-openshift-default-6f85f8ff65-gnstx" Jul 07 16:16:41.380436 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:41.380407 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0aae1a92-e503-48d7-8aa3-a04792622460-istio-envoy\") pod \"e2e-api-b-b728d9-openshift-default-6f85f8ff65-gnstx\" (UID: \"0aae1a92-e503-48d7-8aa3-a04792622460\") " pod="openshift-ingress/e2e-api-b-b728d9-openshift-default-6f85f8ff65-gnstx" Jul 07 16:16:41.380736 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:41.380713 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0aae1a92-e503-48d7-8aa3-a04792622460-istio-podinfo\") pod \"e2e-api-b-b728d9-openshift-default-6f85f8ff65-gnstx\" (UID: \"0aae1a92-e503-48d7-8aa3-a04792622460\") " pod="openshift-ingress/e2e-api-b-b728d9-openshift-default-6f85f8ff65-gnstx" Jul 07 16:16:41.387456 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:41.387427 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0aae1a92-e503-48d7-8aa3-a04792622460-istio-token\") pod \"e2e-api-b-b728d9-openshift-default-6f85f8ff65-gnstx\" (UID: \"0aae1a92-e503-48d7-8aa3-a04792622460\") " pod="openshift-ingress/e2e-api-b-b728d9-openshift-default-6f85f8ff65-gnstx" Jul 07 16:16:41.387571 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:41.387537 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rl878\" (UniqueName: \"kubernetes.io/projected/0aae1a92-e503-48d7-8aa3-a04792622460-kube-api-access-rl878\") pod \"e2e-api-b-b728d9-openshift-default-6f85f8ff65-gnstx\" (UID: \"0aae1a92-e503-48d7-8aa3-a04792622460\") " pod="openshift-ingress/e2e-api-b-b728d9-openshift-default-6f85f8ff65-gnstx" Jul 07 16:16:41.413877 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:41.413847 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:16:41.501688 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:41.501585 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-b728d9-openshift-default-6f85f8ff65-gnstx" Jul 07 16:16:41.683742 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:41.683255 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-b728d9-openshift-default-6f85f8ff65-gnstx"] Jul 07 16:16:41.687815 ip-10-0-132-65 kubenswrapper[2565]: W0707 16:16:41.687782 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0aae1a92_e503_48d7_8aa3_a04792622460.slice/crio-fe07fd5bd679355e6a6b615d4332a629093fbc440245ef4ebf5a8f1d97674cc7 WatchSource:0}: Error finding container fe07fd5bd679355e6a6b615d4332a629093fbc440245ef4ebf5a8f1d97674cc7: Status 404 returned error can't find the container with id fe07fd5bd679355e6a6b615d4332a629093fbc440245ef4ebf5a8f1d97674cc7 Jul 07 16:16:41.690451 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:41.690419 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 07 16:16:41.690538 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:41.690520 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 07 16:16:41.690589 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:41.690557 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 07 16:16:41.888474 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:41.888423 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-b728d9-openshift-default-6f85f8ff65-gnstx" event={"ID":"0aae1a92-e503-48d7-8aa3-a04792622460","Type":"ContainerStarted","Data":"a397efe60f816704df220872baf867090fb9df9737e524bfff5c172d42de33a6"} Jul 07 16:16:41.888901 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:41.888519 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-b728d9-openshift-default-6f85f8ff65-gnstx" event={"ID":"0aae1a92-e503-48d7-8aa3-a04792622460","Type":"ContainerStarted","Data":"fe07fd5bd679355e6a6b615d4332a629093fbc440245ef4ebf5a8f1d97674cc7"} Jul 07 16:16:41.923550 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:41.923483 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-b728d9-openshift-default-6f85f8ff65-gnstx" podStartSLOduration=0.923466854 podStartE2EDuration="923.466854ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:16:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:16:41.920959706 +0000 UTC m=+2140.877537174" watchObservedRunningTime="2026-07-07 16:16:41.923466854 +0000 UTC m=+2140.880044322" Jul 07 16:16:42.501883 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:42.501854 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-b728d9-openshift-default-6f85f8ff65-gnstx" Jul 07 16:16:42.506929 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:42.506905 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-b728d9-openshift-default-6f85f8ff65-gnstx" Jul 07 16:16:42.894553 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:42.894516 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-b728d9-openshift-default-6f85f8ff65-gnstx" Jul 07 16:16:42.895651 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:42.895629 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-b728d9-openshift-default-6f85f8ff65-gnstx" Jul 07 16:16:43.569345 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:43.569271 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:16:43.937983 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:43.937896 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:16:44.032536 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:44.032477 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:16:44.114245 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:44.114204 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:16:44.221554 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:44.221446 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:16:44.807126 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:44.807086 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:16:44.924828 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:44.924798 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:16:48.445277 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:48.445245 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:16:48.930856 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:48.930824 2565 generic.go:356] "Generic (PLEG): container finished" podID="8fe453f9-345b-4288-8e90-ee666c180de9" containerID="f008c76e5c537e94b95c42933ff2976ff640855e8d455b444ff28121b5f538b3" exitCode=0 Jul 07 16:16:48.931042 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:48.930878 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-78b5af-openshift-default-75c949ffb9-7rvw8" event={"ID":"8fe453f9-345b-4288-8e90-ee666c180de9","Type":"ContainerDied","Data":"f008c76e5c537e94b95c42933ff2976ff640855e8d455b444ff28121b5f538b3"} Jul 07 16:16:48.931273 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:48.931258 2565 scope.go:117] "RemoveContainer" containerID="f008c76e5c537e94b95c42933ff2976ff640855e8d455b444ff28121b5f538b3" Jul 07 16:16:48.932322 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:48.932291 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 07 16:16:48.932378 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:48.932364 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 07 16:16:48.932415 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:48.932395 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 07 16:16:49.937306 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:49.937274 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-78b5af-openshift-default-75c949ffb9-7rvw8" event={"ID":"8fe453f9-345b-4288-8e90-ee666c180de9","Type":"ContainerStarted","Data":"135706f9e9f6d9164425e8caca485a758a8cfdfec5e147d53ffacf0663ea6f6b"} Jul 07 16:16:50.850987 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:50.850948 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-78b5af-openshift-default-75c949ffb9-7rvw8" Jul 07 16:16:50.861379 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:50.861332 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-78b5af-openshift-default-75c949ffb9-7rvw8" Jul 07 16:16:50.862868 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:50.862833 2565 patch_prober.go:28] interesting pod/e2e-api-a-78b5af-openshift-default-75c949ffb9-7rvw8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Jul 07 16:16:50.862992 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:50.862910 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-78b5af-openshift-default-75c949ffb9-7rvw8" podUID="8fe453f9-345b-4288-8e90-ee666c180de9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Jul 07 16:16:51.860787 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:51.860752 2565 patch_prober.go:28] interesting pod/e2e-api-a-78b5af-openshift-default-75c949ffb9-7rvw8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Jul 07 16:16:51.861420 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:51.861383 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-78b5af-openshift-default-75c949ffb9-7rvw8" podUID="8fe453f9-345b-4288-8e90-ee666c180de9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Jul 07 16:16:52.861422 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:52.861371 2565 patch_prober.go:28] interesting pod/e2e-api-a-78b5af-openshift-default-75c949ffb9-7rvw8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Jul 07 16:16:52.861952 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:52.861449 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-78b5af-openshift-default-75c949ffb9-7rvw8" podUID="8fe453f9-345b-4288-8e90-ee666c180de9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Jul 07 16:16:53.870949 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:53.870859 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-78b5af-openshift-default-75c949ffb9-7rvw8" Jul 07 16:16:53.874642 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:53.874617 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-78b5af-openshift-default-75c949ffb9-7rvw8" Jul 07 16:16:56.705141 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:56.705107 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:16:57.115207 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:57.115129 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:16:57.501213 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:57.501091 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:16:57.620135 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:57.620108 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:16:57.954598 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:57.954559 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:16:58.936692 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:58.936652 2565 patch_prober.go:28] interesting pod/e2e-api-b-b728d9-openshift-default-6f85f8ff65-gnstx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 07 16:16:58.936893 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:58.936725 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-b728d9-openshift-default-6f85f8ff65-gnstx" podUID="0aae1a92-e503-48d7-8aa3-a04792622460" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 07 16:16:59.995549 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:59.995431 2565 generic.go:356] "Generic (PLEG): container finished" podID="0aae1a92-e503-48d7-8aa3-a04792622460" containerID="a397efe60f816704df220872baf867090fb9df9737e524bfff5c172d42de33a6" exitCode=0 Jul 07 16:16:59.995549 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:59.995517 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-b728d9-openshift-default-6f85f8ff65-gnstx" event={"ID":"0aae1a92-e503-48d7-8aa3-a04792622460","Type":"ContainerDied","Data":"a397efe60f816704df220872baf867090fb9df9737e524bfff5c172d42de33a6"} Jul 07 16:16:59.995980 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:59.995908 2565 scope.go:117] "RemoveContainer" containerID="a397efe60f816704df220872baf867090fb9df9737e524bfff5c172d42de33a6" Jul 07 16:16:59.996941 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:59.996910 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 07 16:16:59.997034 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:59.996984 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 07 16:16:59.997034 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:16:59.997015 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 07 16:17:01.001936 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:01.001895 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-b728d9-openshift-default-6f85f8ff65-gnstx" event={"ID":"0aae1a92-e503-48d7-8aa3-a04792622460","Type":"ContainerStarted","Data":"97b9297d1024ff75e13577bcbdac0cde291cc55a522dd6f39655bfd390f170a6"} Jul 07 16:17:01.502763 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:01.502733 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-b728d9-openshift-default-6f85f8ff65-gnstx" Jul 07 16:17:01.503007 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:01.502984 2565 patch_prober.go:28] interesting pod/e2e-api-b-b728d9-openshift-default-6f85f8ff65-gnstx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Jul 07 16:17:01.503074 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:01.503033 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-b728d9-openshift-default-6f85f8ff65-gnstx" podUID="0aae1a92-e503-48d7-8aa3-a04792622460" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Jul 07 16:17:02.502110 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:02.502065 2565 patch_prober.go:28] interesting pod/e2e-api-b-b728d9-openshift-default-6f85f8ff65-gnstx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Jul 07 16:17:02.502528 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:02.502137 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-b728d9-openshift-default-6f85f8ff65-gnstx" podUID="0aae1a92-e503-48d7-8aa3-a04792622460" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Jul 07 16:17:03.502485 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:03.502449 2565 patch_prober.go:28] interesting pod/e2e-api-b-b728d9-openshift-default-6f85f8ff65-gnstx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Jul 07 16:17:03.502908 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:03.502533 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-b728d9-openshift-default-6f85f8ff65-gnstx" podUID="0aae1a92-e503-48d7-8aa3-a04792622460" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Jul 07 16:17:04.502057 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:04.502019 2565 patch_prober.go:28] interesting pod/e2e-api-b-b728d9-openshift-default-6f85f8ff65-gnstx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Jul 07 16:17:04.502255 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:04.502088 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-b728d9-openshift-default-6f85f8ff65-gnstx" podUID="0aae1a92-e503-48d7-8aa3-a04792622460" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Jul 07 16:17:05.086391 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:05.086353 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:17:05.176846 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:05.176817 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:17:05.434175 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:05.434083 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-78b5af-openshift-default-75c949ffb9-7rvw8"] Jul 07 16:17:05.434350 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:05.434321 2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-78b5af-openshift-default-75c949ffb9-7rvw8" podUID="8fe453f9-345b-4288-8e90-ee666c180de9" containerName="istio-proxy" containerID="cri-o://135706f9e9f6d9164425e8caca485a758a8cfdfec5e147d53ffacf0663ea6f6b" gracePeriod=30 Jul 07 16:17:05.457507 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:05.457468 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:17:05.505681 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:05.505640 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-b728d9-openshift-default-6f85f8ff65-gnstx" Jul 07 16:17:05.506163 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:05.506144 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-b728d9-openshift-default-6f85f8ff65-gnstx" Jul 07 16:17:05.506851 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:05.506834 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-b728d9-openshift-default-6f85f8ff65-gnstx" Jul 07 16:17:05.690897 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:05.690809 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:17:05.910082 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:05.910051 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:17:06.075589 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:06.075551 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:17:06.191002 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:06.190969 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:17:06.287152 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:06.287115 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:17:06.616636 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:06.616603 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:17:08.875060 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:08.875016 2565 patch_prober.go:28] interesting pod/e2e-api-a-78b5af-openshift-default-75c949ffb9-7rvw8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 07 16:17:08.875530 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:08.875109 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-78b5af-openshift-default-75c949ffb9-7rvw8" podUID="8fe453f9-345b-4288-8e90-ee666c180de9" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 07 16:17:10.701051 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:10.700991 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-78b5af-openshift-default-75c949ffb9-7rvw8" Jul 07 16:17:10.762826 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:10.762792 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8fe453f9-345b-4288-8e90-ee666c180de9-istio-data\") pod \"8fe453f9-345b-4288-8e90-ee666c180de9\" (UID: \"8fe453f9-345b-4288-8e90-ee666c180de9\") " Jul 07 16:17:10.762826 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:10.762830 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wfjlc\" (UniqueName: \"kubernetes.io/projected/8fe453f9-345b-4288-8e90-ee666c180de9-kube-api-access-wfjlc\") pod \"8fe453f9-345b-4288-8e90-ee666c180de9\" (UID: \"8fe453f9-345b-4288-8e90-ee666c180de9\") " Jul 07 16:17:10.763091 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:10.762851 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fe453f9-345b-4288-8e90-ee666c180de9-credential-socket\") pod \"8fe453f9-345b-4288-8e90-ee666c180de9\" (UID: \"8fe453f9-345b-4288-8e90-ee666c180de9\") " Jul 07 16:17:10.763091 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:10.762894 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8fe453f9-345b-4288-8e90-ee666c180de9-workload-certs\") pod \"8fe453f9-345b-4288-8e90-ee666c180de9\" (UID: \"8fe453f9-345b-4288-8e90-ee666c180de9\") " Jul 07 16:17:10.763091 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:10.762924 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8fe453f9-345b-4288-8e90-ee666c180de9-istio-podinfo\") pod \"8fe453f9-345b-4288-8e90-ee666c180de9\" (UID: \"8fe453f9-345b-4288-8e90-ee666c180de9\") " Jul 07 16:17:10.763091 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:10.762951 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8fe453f9-345b-4288-8e90-ee666c180de9-istio-envoy\") pod \"8fe453f9-345b-4288-8e90-ee666c180de9\" (UID: \"8fe453f9-345b-4288-8e90-ee666c180de9\") " Jul 07 16:17:10.763091 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:10.762983 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8fe453f9-345b-4288-8e90-ee666c180de9-istiod-ca-cert\") pod \"8fe453f9-345b-4288-8e90-ee666c180de9\" (UID: \"8fe453f9-345b-4288-8e90-ee666c180de9\") " Jul 07 16:17:10.763352 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:10.763165 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fe453f9-345b-4288-8e90-ee666c180de9-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "8fe453f9-345b-4288-8e90-ee666c180de9" (UID: "8fe453f9-345b-4288-8e90-ee666c180de9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 16:17:10.763352 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:10.763225 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fe453f9-345b-4288-8e90-ee666c180de9-workload-socket\") pod \"8fe453f9-345b-4288-8e90-ee666c180de9\" (UID: \"8fe453f9-345b-4288-8e90-ee666c180de9\") " Jul 07 16:17:10.763352 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:10.763343 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8fe453f9-345b-4288-8e90-ee666c180de9-istio-token\") pod \"8fe453f9-345b-4288-8e90-ee666c180de9\" (UID: \"8fe453f9-345b-4288-8e90-ee666c180de9\") " Jul 07 16:17:10.763523 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:10.763459 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fe453f9-345b-4288-8e90-ee666c180de9-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "8fe453f9-345b-4288-8e90-ee666c180de9" (UID: "8fe453f9-345b-4288-8e90-ee666c180de9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 16:17:10.763523 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:10.763484 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8fe453f9-345b-4288-8e90-ee666c180de9-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "8fe453f9-345b-4288-8e90-ee666c180de9" (UID: "8fe453f9-345b-4288-8e90-ee666c180de9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 16:17:10.763633 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:10.763603 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fe453f9-345b-4288-8e90-ee666c180de9-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "8fe453f9-345b-4288-8e90-ee666c180de9" (UID: "8fe453f9-345b-4288-8e90-ee666c180de9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 16:17:10.763801 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:10.763777 2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fe453f9-345b-4288-8e90-ee666c180de9-credential-socket\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 07 16:17:10.763801 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:10.763802 2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8fe453f9-345b-4288-8e90-ee666c180de9-workload-certs\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 07 16:17:10.763980 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:10.763817 2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8fe453f9-345b-4288-8e90-ee666c180de9-istiod-ca-cert\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 07 16:17:10.763980 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:10.763828 2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fe453f9-345b-4288-8e90-ee666c180de9-workload-socket\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 07 16:17:10.763980 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:10.763829 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fe453f9-345b-4288-8e90-ee666c180de9-istio-data" (OuterVolumeSpecName: "istio-data") pod "8fe453f9-345b-4288-8e90-ee666c180de9" (UID: "8fe453f9-345b-4288-8e90-ee666c180de9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 16:17:10.765195 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:10.765173 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fe453f9-345b-4288-8e90-ee666c180de9-kube-api-access-wfjlc" (OuterVolumeSpecName: "kube-api-access-wfjlc") pod "8fe453f9-345b-4288-8e90-ee666c180de9" (UID: "8fe453f9-345b-4288-8e90-ee666c180de9"). InnerVolumeSpecName "kube-api-access-wfjlc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 16:17:10.765465 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:10.765446 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fe453f9-345b-4288-8e90-ee666c180de9-istio-token" (OuterVolumeSpecName: "istio-token") pod "8fe453f9-345b-4288-8e90-ee666c180de9" (UID: "8fe453f9-345b-4288-8e90-ee666c180de9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 16:17:10.765670 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:10.765648 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fe453f9-345b-4288-8e90-ee666c180de9-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "8fe453f9-345b-4288-8e90-ee666c180de9" (UID: "8fe453f9-345b-4288-8e90-ee666c180de9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 16:17:10.765670 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:10.765656 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8fe453f9-345b-4288-8e90-ee666c180de9-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "8fe453f9-345b-4288-8e90-ee666c180de9" (UID: "8fe453f9-345b-4288-8e90-ee666c180de9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 07 16:17:10.864563 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:10.864530 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wfjlc\" (UniqueName: \"kubernetes.io/projected/8fe453f9-345b-4288-8e90-ee666c180de9-kube-api-access-wfjlc\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 07 16:17:10.864563 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:10.864557 2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8fe453f9-345b-4288-8e90-ee666c180de9-istio-podinfo\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 07 16:17:10.864563 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:10.864569 2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8fe453f9-345b-4288-8e90-ee666c180de9-istio-envoy\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 07 16:17:10.864809 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:10.864578 2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8fe453f9-345b-4288-8e90-ee666c180de9-istio-token\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 07 16:17:10.864809 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:10.864585 2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8fe453f9-345b-4288-8e90-ee666c180de9-istio-data\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 07 16:17:11.055674 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:11.055584 2565 generic.go:356] "Generic (PLEG): container finished" podID="8fe453f9-345b-4288-8e90-ee666c180de9" containerID="135706f9e9f6d9164425e8caca485a758a8cfdfec5e147d53ffacf0663ea6f6b" exitCode=0 Jul 07 16:17:11.055674 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:11.055632 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-78b5af-openshift-default-75c949ffb9-7rvw8" Jul 07 16:17:11.055674 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:11.055636 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-78b5af-openshift-default-75c949ffb9-7rvw8" event={"ID":"8fe453f9-345b-4288-8e90-ee666c180de9","Type":"ContainerDied","Data":"135706f9e9f6d9164425e8caca485a758a8cfdfec5e147d53ffacf0663ea6f6b"} Jul 07 16:17:11.055897 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:11.055686 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-78b5af-openshift-default-75c949ffb9-7rvw8" event={"ID":"8fe453f9-345b-4288-8e90-ee666c180de9","Type":"ContainerDied","Data":"98638597a0383c6395212d93ffc072379f292c9152112edf9f24a8c4ba8ee0a0"} Jul 07 16:17:11.055897 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:11.055709 2565 scope.go:117] "RemoveContainer" containerID="135706f9e9f6d9164425e8caca485a758a8cfdfec5e147d53ffacf0663ea6f6b" Jul 07 16:17:11.065673 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:11.065653 2565 scope.go:117] "RemoveContainer" containerID="f008c76e5c537e94b95c42933ff2976ff640855e8d455b444ff28121b5f538b3" Jul 07 16:17:11.073753 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:11.073734 2565 scope.go:117] "RemoveContainer" containerID="135706f9e9f6d9164425e8caca485a758a8cfdfec5e147d53ffacf0663ea6f6b" Jul 07 16:17:11.074021 ip-10-0-132-65 kubenswrapper[2565]: E0707 16:17:11.074000 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"135706f9e9f6d9164425e8caca485a758a8cfdfec5e147d53ffacf0663ea6f6b\": container with ID starting with 135706f9e9f6d9164425e8caca485a758a8cfdfec5e147d53ffacf0663ea6f6b not found: ID does not exist" containerID="135706f9e9f6d9164425e8caca485a758a8cfdfec5e147d53ffacf0663ea6f6b" Jul 07 16:17:11.074095 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:11.074035 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"135706f9e9f6d9164425e8caca485a758a8cfdfec5e147d53ffacf0663ea6f6b"} err="failed to get container status \"135706f9e9f6d9164425e8caca485a758a8cfdfec5e147d53ffacf0663ea6f6b\": rpc error: code = NotFound desc = could not find container \"135706f9e9f6d9164425e8caca485a758a8cfdfec5e147d53ffacf0663ea6f6b\": container with ID starting with 135706f9e9f6d9164425e8caca485a758a8cfdfec5e147d53ffacf0663ea6f6b not found: ID does not exist" Jul 07 16:17:11.074095 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:11.074059 2565 scope.go:117] "RemoveContainer" containerID="f008c76e5c537e94b95c42933ff2976ff640855e8d455b444ff28121b5f538b3" Jul 07 16:17:11.074309 ip-10-0-132-65 kubenswrapper[2565]: E0707 16:17:11.074292 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f008c76e5c537e94b95c42933ff2976ff640855e8d455b444ff28121b5f538b3\": container with ID starting with f008c76e5c537e94b95c42933ff2976ff640855e8d455b444ff28121b5f538b3 not found: ID does not exist" containerID="f008c76e5c537e94b95c42933ff2976ff640855e8d455b444ff28121b5f538b3" Jul 07 16:17:11.074349 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:11.074316 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f008c76e5c537e94b95c42933ff2976ff640855e8d455b444ff28121b5f538b3"} err="failed to get container status \"f008c76e5c537e94b95c42933ff2976ff640855e8d455b444ff28121b5f538b3\": rpc error: code = NotFound desc = could not find container \"f008c76e5c537e94b95c42933ff2976ff640855e8d455b444ff28121b5f538b3\": container with ID starting with f008c76e5c537e94b95c42933ff2976ff640855e8d455b444ff28121b5f538b3 not found: ID does not exist" Jul 07 16:17:11.083570 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:11.083547 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-78b5af-openshift-default-75c949ffb9-7rvw8"] Jul 07 16:17:11.088748 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:11.088726 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-78b5af-openshift-default-75c949ffb9-7rvw8"] Jul 07 16:17:11.661342 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:11.661308 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8fe453f9-345b-4288-8e90-ee666c180de9" path="/var/lib/kubelet/pods/8fe453f9-345b-4288-8e90-ee666c180de9/volumes" Jul 07 16:17:13.091872 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:13.091819 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:17:13.189784 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:13.189742 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:17:13.301117 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:13.301087 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-b728d9-openshift-default-6f85f8ff65-gnstx"] Jul 07 16:17:13.301295 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:13.301274 2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-b728d9-openshift-default-6f85f8ff65-gnstx" podUID="0aae1a92-e503-48d7-8aa3-a04792622460" containerName="istio-proxy" containerID="cri-o://97b9297d1024ff75e13577bcbdac0cde291cc55a522dd6f39655bfd390f170a6" gracePeriod=30 Jul 07 16:17:13.351802 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:13.351719 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:17:13.582017 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:13.581982 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:17:13.917268 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:13.917230 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:17:14.258909 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:14.258821 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-382396-openshift-default-66d7949b4c-qq7rq"] Jul 07 16:17:14.259368 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:14.259230 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8fe453f9-345b-4288-8e90-ee666c180de9" containerName="istio-proxy" Jul 07 16:17:14.259368 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:14.259243 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="8fe453f9-345b-4288-8e90-ee666c180de9" containerName="istio-proxy" Jul 07 16:17:14.259368 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:14.259259 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="13c0936e-d449-4b9b-882e-587993a0df6f" containerName="istio-proxy" Jul 07 16:17:14.259368 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:14.259264 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="13c0936e-d449-4b9b-882e-587993a0df6f" containerName="istio-proxy" Jul 07 16:17:14.259368 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:14.259323 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="8fe453f9-345b-4288-8e90-ee666c180de9" containerName="istio-proxy" Jul 07 16:17:14.262887 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:14.262861 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-382396-openshift-default-66d7949b4c-qq7rq" Jul 07 16:17:14.267632 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:14.267607 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-382396-openshift-default-dockercfg-7kkrd\"" Jul 07 16:17:14.285459 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:14.285436 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-382396-openshift-default-66d7949b4c-qq7rq"] Jul 07 16:17:14.294080 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:14.294057 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0094cc87-2dc6-4827-8288-a3406bf56018-workload-certs\") pod \"e2e-shared-a-382396-openshift-default-66d7949b4c-qq7rq\" (UID: \"0094cc87-2dc6-4827-8288-a3406bf56018\") " pod="openshift-ingress/e2e-shared-a-382396-openshift-default-66d7949b4c-qq7rq" Jul 07 16:17:14.294212 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:14.294127 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0094cc87-2dc6-4827-8288-a3406bf56018-istio-podinfo\") pod \"e2e-shared-a-382396-openshift-default-66d7949b4c-qq7rq\" (UID: \"0094cc87-2dc6-4827-8288-a3406bf56018\") " pod="openshift-ingress/e2e-shared-a-382396-openshift-default-66d7949b4c-qq7rq" Jul 07 16:17:14.294212 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:14.294154 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0094cc87-2dc6-4827-8288-a3406bf56018-istio-token\") pod \"e2e-shared-a-382396-openshift-default-66d7949b4c-qq7rq\" (UID: \"0094cc87-2dc6-4827-8288-a3406bf56018\") " pod="openshift-ingress/e2e-shared-a-382396-openshift-default-66d7949b4c-qq7rq" Jul 07 16:17:14.294212 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:14.294177 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0094cc87-2dc6-4827-8288-a3406bf56018-istio-envoy\") pod \"e2e-shared-a-382396-openshift-default-66d7949b4c-qq7rq\" (UID: \"0094cc87-2dc6-4827-8288-a3406bf56018\") " pod="openshift-ingress/e2e-shared-a-382396-openshift-default-66d7949b4c-qq7rq" Jul 07 16:17:14.294395 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:14.294249 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0094cc87-2dc6-4827-8288-a3406bf56018-istio-data\") pod \"e2e-shared-a-382396-openshift-default-66d7949b4c-qq7rq\" (UID: \"0094cc87-2dc6-4827-8288-a3406bf56018\") " pod="openshift-ingress/e2e-shared-a-382396-openshift-default-66d7949b4c-qq7rq" Jul 07 16:17:14.294395 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:14.294273 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0094cc87-2dc6-4827-8288-a3406bf56018-istiod-ca-cert\") pod \"e2e-shared-a-382396-openshift-default-66d7949b4c-qq7rq\" (UID: \"0094cc87-2dc6-4827-8288-a3406bf56018\") " pod="openshift-ingress/e2e-shared-a-382396-openshift-default-66d7949b4c-qq7rq" Jul 07 16:17:14.294395 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:14.294321 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jpn88\" (UniqueName: \"kubernetes.io/projected/0094cc87-2dc6-4827-8288-a3406bf56018-kube-api-access-jpn88\") pod \"e2e-shared-a-382396-openshift-default-66d7949b4c-qq7rq\" (UID: \"0094cc87-2dc6-4827-8288-a3406bf56018\") " pod="openshift-ingress/e2e-shared-a-382396-openshift-default-66d7949b4c-qq7rq" Jul 07 16:17:14.294395 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:14.294347 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0094cc87-2dc6-4827-8288-a3406bf56018-workload-socket\") pod \"e2e-shared-a-382396-openshift-default-66d7949b4c-qq7rq\" (UID: \"0094cc87-2dc6-4827-8288-a3406bf56018\") " pod="openshift-ingress/e2e-shared-a-382396-openshift-default-66d7949b4c-qq7rq" Jul 07 16:17:14.294395 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:14.294372 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0094cc87-2dc6-4827-8288-a3406bf56018-credential-socket\") pod \"e2e-shared-a-382396-openshift-default-66d7949b4c-qq7rq\" (UID: \"0094cc87-2dc6-4827-8288-a3406bf56018\") " pod="openshift-ingress/e2e-shared-a-382396-openshift-default-66d7949b4c-qq7rq" Jul 07 16:17:14.297211 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:14.297189 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:17:14.395254 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:14.395214 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0094cc87-2dc6-4827-8288-a3406bf56018-istio-podinfo\") pod \"e2e-shared-a-382396-openshift-default-66d7949b4c-qq7rq\" (UID: \"0094cc87-2dc6-4827-8288-a3406bf56018\") " pod="openshift-ingress/e2e-shared-a-382396-openshift-default-66d7949b4c-qq7rq" Jul 07 16:17:14.395450 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:14.395259 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0094cc87-2dc6-4827-8288-a3406bf56018-istio-token\") pod \"e2e-shared-a-382396-openshift-default-66d7949b4c-qq7rq\" (UID: \"0094cc87-2dc6-4827-8288-a3406bf56018\") " pod="openshift-ingress/e2e-shared-a-382396-openshift-default-66d7949b4c-qq7rq" Jul 07 16:17:14.395450 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:14.395296 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0094cc87-2dc6-4827-8288-a3406bf56018-istio-envoy\") pod \"e2e-shared-a-382396-openshift-default-66d7949b4c-qq7rq\" (UID: \"0094cc87-2dc6-4827-8288-a3406bf56018\") " pod="openshift-ingress/e2e-shared-a-382396-openshift-default-66d7949b4c-qq7rq" Jul 07 16:17:14.395450 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:14.395433 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0094cc87-2dc6-4827-8288-a3406bf56018-istio-data\") pod \"e2e-shared-a-382396-openshift-default-66d7949b4c-qq7rq\" (UID: \"0094cc87-2dc6-4827-8288-a3406bf56018\") " pod="openshift-ingress/e2e-shared-a-382396-openshift-default-66d7949b4c-qq7rq" Jul 07 16:17:14.395667 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:14.395476 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0094cc87-2dc6-4827-8288-a3406bf56018-istiod-ca-cert\") pod \"e2e-shared-a-382396-openshift-default-66d7949b4c-qq7rq\" (UID: \"0094cc87-2dc6-4827-8288-a3406bf56018\") " pod="openshift-ingress/e2e-shared-a-382396-openshift-default-66d7949b4c-qq7rq" Jul 07 16:17:14.395667 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:14.395558 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jpn88\" (UniqueName: \"kubernetes.io/projected/0094cc87-2dc6-4827-8288-a3406bf56018-kube-api-access-jpn88\") pod \"e2e-shared-a-382396-openshift-default-66d7949b4c-qq7rq\" (UID: \"0094cc87-2dc6-4827-8288-a3406bf56018\") " pod="openshift-ingress/e2e-shared-a-382396-openshift-default-66d7949b4c-qq7rq" Jul 07 16:17:14.395667 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:14.395584 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0094cc87-2dc6-4827-8288-a3406bf56018-workload-socket\") pod \"e2e-shared-a-382396-openshift-default-66d7949b4c-qq7rq\" (UID: \"0094cc87-2dc6-4827-8288-a3406bf56018\") " pod="openshift-ingress/e2e-shared-a-382396-openshift-default-66d7949b4c-qq7rq" Jul 07 16:17:14.395667 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:14.395619 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0094cc87-2dc6-4827-8288-a3406bf56018-credential-socket\") pod \"e2e-shared-a-382396-openshift-default-66d7949b4c-qq7rq\" (UID: \"0094cc87-2dc6-4827-8288-a3406bf56018\") " pod="openshift-ingress/e2e-shared-a-382396-openshift-default-66d7949b4c-qq7rq" Jul 07 16:17:14.395667 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:14.395652 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0094cc87-2dc6-4827-8288-a3406bf56018-workload-certs\") pod \"e2e-shared-a-382396-openshift-default-66d7949b4c-qq7rq\" (UID: \"0094cc87-2dc6-4827-8288-a3406bf56018\") " pod="openshift-ingress/e2e-shared-a-382396-openshift-default-66d7949b4c-qq7rq" Jul 07 16:17:14.396004 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:14.395982 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0094cc87-2dc6-4827-8288-a3406bf56018-workload-certs\") pod \"e2e-shared-a-382396-openshift-default-66d7949b4c-qq7rq\" (UID: \"0094cc87-2dc6-4827-8288-a3406bf56018\") " pod="openshift-ingress/e2e-shared-a-382396-openshift-default-66d7949b4c-qq7rq" Jul 07 16:17:14.396172 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:14.396127 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0094cc87-2dc6-4827-8288-a3406bf56018-credential-socket\") pod \"e2e-shared-a-382396-openshift-default-66d7949b4c-qq7rq\" (UID: \"0094cc87-2dc6-4827-8288-a3406bf56018\") " pod="openshift-ingress/e2e-shared-a-382396-openshift-default-66d7949b4c-qq7rq" Jul 07 16:17:14.396172 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:14.396038 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0094cc87-2dc6-4827-8288-a3406bf56018-workload-socket\") pod \"e2e-shared-a-382396-openshift-default-66d7949b4c-qq7rq\" (UID: \"0094cc87-2dc6-4827-8288-a3406bf56018\") " pod="openshift-ingress/e2e-shared-a-382396-openshift-default-66d7949b4c-qq7rq" Jul 07 16:17:14.396353 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:14.396275 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0094cc87-2dc6-4827-8288-a3406bf56018-istiod-ca-cert\") pod \"e2e-shared-a-382396-openshift-default-66d7949b4c-qq7rq\" (UID: \"0094cc87-2dc6-4827-8288-a3406bf56018\") " pod="openshift-ingress/e2e-shared-a-382396-openshift-default-66d7949b4c-qq7rq" Jul 07 16:17:14.396399 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:14.396385 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0094cc87-2dc6-4827-8288-a3406bf56018-istio-data\") pod \"e2e-shared-a-382396-openshift-default-66d7949b4c-qq7rq\" (UID: \"0094cc87-2dc6-4827-8288-a3406bf56018\") " pod="openshift-ingress/e2e-shared-a-382396-openshift-default-66d7949b4c-qq7rq" Jul 07 16:17:14.397724 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:14.397706 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0094cc87-2dc6-4827-8288-a3406bf56018-istio-envoy\") pod \"e2e-shared-a-382396-openshift-default-66d7949b4c-qq7rq\" (UID: \"0094cc87-2dc6-4827-8288-a3406bf56018\") " pod="openshift-ingress/e2e-shared-a-382396-openshift-default-66d7949b4c-qq7rq" Jul 07 16:17:14.397933 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:14.397915 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0094cc87-2dc6-4827-8288-a3406bf56018-istio-podinfo\") pod \"e2e-shared-a-382396-openshift-default-66d7949b4c-qq7rq\" (UID: \"0094cc87-2dc6-4827-8288-a3406bf56018\") " pod="openshift-ingress/e2e-shared-a-382396-openshift-default-66d7949b4c-qq7rq" Jul 07 16:17:14.407168 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:14.407138 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0094cc87-2dc6-4827-8288-a3406bf56018-istio-token\") pod \"e2e-shared-a-382396-openshift-default-66d7949b4c-qq7rq\" (UID: \"0094cc87-2dc6-4827-8288-a3406bf56018\") " pod="openshift-ingress/e2e-shared-a-382396-openshift-default-66d7949b4c-qq7rq" Jul 07 16:17:14.407330 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:14.407309 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jpn88\" (UniqueName: \"kubernetes.io/projected/0094cc87-2dc6-4827-8288-a3406bf56018-kube-api-access-jpn88\") pod \"e2e-shared-a-382396-openshift-default-66d7949b4c-qq7rq\" (UID: \"0094cc87-2dc6-4827-8288-a3406bf56018\") " pod="openshift-ingress/e2e-shared-a-382396-openshift-default-66d7949b4c-qq7rq" Jul 07 16:17:14.434983 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:14.434935 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:17:14.573382 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:14.573344 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-382396-openshift-default-66d7949b4c-qq7rq" Jul 07 16:17:14.727089 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:14.727067 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-382396-openshift-default-66d7949b4c-qq7rq"] Jul 07 16:17:14.728789 ip-10-0-132-65 kubenswrapper[2565]: W0707 16:17:14.728761 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0094cc87_2dc6_4827_8288_a3406bf56018.slice/crio-97d1f7202068a79f46b14df25e67b112ae38f001116039b027a15b3a86f1a18f WatchSource:0}: Error finding container 97d1f7202068a79f46b14df25e67b112ae38f001116039b027a15b3a86f1a18f: Status 404 returned error can't find the container with id 97d1f7202068a79f46b14df25e67b112ae38f001116039b027a15b3a86f1a18f Jul 07 16:17:14.730971 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:14.730944 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 07 16:17:14.731029 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:14.731005 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 07 16:17:14.731068 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:14.731033 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 07 16:17:15.076080 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:15.076045 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-382396-openshift-default-66d7949b4c-qq7rq" event={"ID":"0094cc87-2dc6-4827-8288-a3406bf56018","Type":"ContainerStarted","Data":"73f802b80f728d9a02c263972746bca5d3d091b4c8ab630f6f07c491d3abeafe"} Jul 07 16:17:15.076236 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:15.076102 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-382396-openshift-default-66d7949b4c-qq7rq" event={"ID":"0094cc87-2dc6-4827-8288-a3406bf56018","Type":"ContainerStarted","Data":"97d1f7202068a79f46b14df25e67b112ae38f001116039b027a15b3a86f1a18f"} Jul 07 16:17:15.105798 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:15.105750 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-382396-openshift-default-66d7949b4c-qq7rq" podStartSLOduration=1.105734018 podStartE2EDuration="1.105734018s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:17:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:17:15.102933606 +0000 UTC m=+2174.059511073" watchObservedRunningTime="2026-07-07 16:17:15.105734018 +0000 UTC m=+2174.062311483" Jul 07 16:17:15.574333 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:15.574299 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-382396-openshift-default-66d7949b4c-qq7rq" Jul 07 16:17:15.579323 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:15.579296 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-382396-openshift-default-66d7949b4c-qq7rq" Jul 07 16:17:16.081291 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:16.081261 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-382396-openshift-default-66d7949b4c-qq7rq" Jul 07 16:17:16.082273 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:16.082254 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-382396-openshift-default-66d7949b4c-qq7rq" Jul 07 16:17:17.221724 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:17.221685 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:17:17.310290 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:17.310259 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:17:17.413301 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:17.413270 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:17:17.909088 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:17.909057 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:17:18.038728 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:18.038696 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:17:18.128030 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:18.128000 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:17:18.561549 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:18.561473 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-b728d9-openshift-default-6f85f8ff65-gnstx" Jul 07 16:17:18.633561 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:18.633529 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0aae1a92-e503-48d7-8aa3-a04792622460-istio-data\") pod \"0aae1a92-e503-48d7-8aa3-a04792622460\" (UID: \"0aae1a92-e503-48d7-8aa3-a04792622460\") " Jul 07 16:17:18.633561 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:18.633564 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0aae1a92-e503-48d7-8aa3-a04792622460-credential-socket\") pod \"0aae1a92-e503-48d7-8aa3-a04792622460\" (UID: \"0aae1a92-e503-48d7-8aa3-a04792622460\") " Jul 07 16:17:18.633807 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:18.633587 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0aae1a92-e503-48d7-8aa3-a04792622460-istiod-ca-cert\") pod \"0aae1a92-e503-48d7-8aa3-a04792622460\" (UID: \"0aae1a92-e503-48d7-8aa3-a04792622460\") " Jul 07 16:17:18.633807 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:18.633613 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0aae1a92-e503-48d7-8aa3-a04792622460-workload-certs\") pod \"0aae1a92-e503-48d7-8aa3-a04792622460\" (UID: \"0aae1a92-e503-48d7-8aa3-a04792622460\") " Jul 07 16:17:18.633807 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:18.633655 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0aae1a92-e503-48d7-8aa3-a04792622460-istio-podinfo\") pod \"0aae1a92-e503-48d7-8aa3-a04792622460\" (UID: \"0aae1a92-e503-48d7-8aa3-a04792622460\") " Jul 07 16:17:18.633807 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:18.633701 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0aae1a92-e503-48d7-8aa3-a04792622460-istio-token\") pod \"0aae1a92-e503-48d7-8aa3-a04792622460\" (UID: \"0aae1a92-e503-48d7-8aa3-a04792622460\") " Jul 07 16:17:18.633807 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:18.633735 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0aae1a92-e503-48d7-8aa3-a04792622460-istio-envoy\") pod \"0aae1a92-e503-48d7-8aa3-a04792622460\" (UID: \"0aae1a92-e503-48d7-8aa3-a04792622460\") " Jul 07 16:17:18.633807 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:18.633775 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0aae1a92-e503-48d7-8aa3-a04792622460-workload-socket\") pod \"0aae1a92-e503-48d7-8aa3-a04792622460\" (UID: \"0aae1a92-e503-48d7-8aa3-a04792622460\") " Jul 07 16:17:18.634117 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:18.633872 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rl878\" (UniqueName: \"kubernetes.io/projected/0aae1a92-e503-48d7-8aa3-a04792622460-kube-api-access-rl878\") pod \"0aae1a92-e503-48d7-8aa3-a04792622460\" (UID: \"0aae1a92-e503-48d7-8aa3-a04792622460\") " Jul 07 16:17:18.634117 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:18.633874 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0aae1a92-e503-48d7-8aa3-a04792622460-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "0aae1a92-e503-48d7-8aa3-a04792622460" (UID: "0aae1a92-e503-48d7-8aa3-a04792622460"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 16:17:18.634117 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:18.634082 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0aae1a92-e503-48d7-8aa3-a04792622460-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "0aae1a92-e503-48d7-8aa3-a04792622460" (UID: "0aae1a92-e503-48d7-8aa3-a04792622460"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 16:17:18.634117 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:18.634084 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0aae1a92-e503-48d7-8aa3-a04792622460-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "0aae1a92-e503-48d7-8aa3-a04792622460" (UID: "0aae1a92-e503-48d7-8aa3-a04792622460"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 16:17:18.634321 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:18.634214 2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0aae1a92-e503-48d7-8aa3-a04792622460-credential-socket\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 07 16:17:18.634321 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:18.634232 2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0aae1a92-e503-48d7-8aa3-a04792622460-istiod-ca-cert\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 07 16:17:18.634321 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:18.634245 2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0aae1a92-e503-48d7-8aa3-a04792622460-workload-certs\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 07 16:17:18.634462 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:18.634360 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0aae1a92-e503-48d7-8aa3-a04792622460-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "0aae1a92-e503-48d7-8aa3-a04792622460" (UID: "0aae1a92-e503-48d7-8aa3-a04792622460"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 16:17:18.634462 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:18.634370 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0aae1a92-e503-48d7-8aa3-a04792622460-istio-data" (OuterVolumeSpecName: "istio-data") pod "0aae1a92-e503-48d7-8aa3-a04792622460" (UID: "0aae1a92-e503-48d7-8aa3-a04792622460"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 16:17:18.636540 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:18.636486 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0aae1a92-e503-48d7-8aa3-a04792622460-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "0aae1a92-e503-48d7-8aa3-a04792622460" (UID: "0aae1a92-e503-48d7-8aa3-a04792622460"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 07 16:17:18.636540 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:18.636522 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0aae1a92-e503-48d7-8aa3-a04792622460-istio-token" (OuterVolumeSpecName: "istio-token") pod "0aae1a92-e503-48d7-8aa3-a04792622460" (UID: "0aae1a92-e503-48d7-8aa3-a04792622460"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 16:17:18.636686 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:18.636610 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0aae1a92-e503-48d7-8aa3-a04792622460-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "0aae1a92-e503-48d7-8aa3-a04792622460" (UID: "0aae1a92-e503-48d7-8aa3-a04792622460"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 16:17:18.636686 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:18.636620 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0aae1a92-e503-48d7-8aa3-a04792622460-kube-api-access-rl878" (OuterVolumeSpecName: "kube-api-access-rl878") pod "0aae1a92-e503-48d7-8aa3-a04792622460" (UID: "0aae1a92-e503-48d7-8aa3-a04792622460"). InnerVolumeSpecName "kube-api-access-rl878". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 16:17:18.735635 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:18.735587 2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0aae1a92-e503-48d7-8aa3-a04792622460-workload-socket\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 07 16:17:18.735635 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:18.735622 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rl878\" (UniqueName: \"kubernetes.io/projected/0aae1a92-e503-48d7-8aa3-a04792622460-kube-api-access-rl878\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 07 16:17:18.735635 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:18.735636 2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0aae1a92-e503-48d7-8aa3-a04792622460-istio-data\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 07 16:17:18.735952 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:18.735648 2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0aae1a92-e503-48d7-8aa3-a04792622460-istio-podinfo\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 07 16:17:18.735952 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:18.735661 2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0aae1a92-e503-48d7-8aa3-a04792622460-istio-token\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 07 16:17:18.735952 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:18.735672 2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0aae1a92-e503-48d7-8aa3-a04792622460-istio-envoy\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 07 16:17:19.097928 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:19.097893 2565 generic.go:356] "Generic (PLEG): container finished" podID="0aae1a92-e503-48d7-8aa3-a04792622460" containerID="97b9297d1024ff75e13577bcbdac0cde291cc55a522dd6f39655bfd390f170a6" exitCode=0 Jul 07 16:17:19.098124 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:19.097939 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-b728d9-openshift-default-6f85f8ff65-gnstx" Jul 07 16:17:19.098124 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:19.097959 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-b728d9-openshift-default-6f85f8ff65-gnstx" event={"ID":"0aae1a92-e503-48d7-8aa3-a04792622460","Type":"ContainerDied","Data":"97b9297d1024ff75e13577bcbdac0cde291cc55a522dd6f39655bfd390f170a6"} Jul 07 16:17:19.098124 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:19.098002 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-b728d9-openshift-default-6f85f8ff65-gnstx" event={"ID":"0aae1a92-e503-48d7-8aa3-a04792622460","Type":"ContainerDied","Data":"fe07fd5bd679355e6a6b615d4332a629093fbc440245ef4ebf5a8f1d97674cc7"} Jul 07 16:17:19.098124 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:19.098022 2565 scope.go:117] "RemoveContainer" containerID="97b9297d1024ff75e13577bcbdac0cde291cc55a522dd6f39655bfd390f170a6" Jul 07 16:17:19.107109 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:19.107088 2565 scope.go:117] "RemoveContainer" containerID="a397efe60f816704df220872baf867090fb9df9737e524bfff5c172d42de33a6" Jul 07 16:17:19.115341 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:19.115323 2565 scope.go:117] "RemoveContainer" containerID="97b9297d1024ff75e13577bcbdac0cde291cc55a522dd6f39655bfd390f170a6" Jul 07 16:17:19.115686 ip-10-0-132-65 kubenswrapper[2565]: E0707 16:17:19.115663 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97b9297d1024ff75e13577bcbdac0cde291cc55a522dd6f39655bfd390f170a6\": container with ID starting with 97b9297d1024ff75e13577bcbdac0cde291cc55a522dd6f39655bfd390f170a6 not found: ID does not exist" containerID="97b9297d1024ff75e13577bcbdac0cde291cc55a522dd6f39655bfd390f170a6" Jul 07 16:17:19.115778 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:19.115707 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97b9297d1024ff75e13577bcbdac0cde291cc55a522dd6f39655bfd390f170a6"} err="failed to get container status \"97b9297d1024ff75e13577bcbdac0cde291cc55a522dd6f39655bfd390f170a6\": rpc error: code = NotFound desc = could not find container \"97b9297d1024ff75e13577bcbdac0cde291cc55a522dd6f39655bfd390f170a6\": container with ID starting with 97b9297d1024ff75e13577bcbdac0cde291cc55a522dd6f39655bfd390f170a6 not found: ID does not exist" Jul 07 16:17:19.115778 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:19.115724 2565 scope.go:117] "RemoveContainer" containerID="a397efe60f816704df220872baf867090fb9df9737e524bfff5c172d42de33a6" Jul 07 16:17:19.115968 ip-10-0-132-65 kubenswrapper[2565]: E0707 16:17:19.115950 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a397efe60f816704df220872baf867090fb9df9737e524bfff5c172d42de33a6\": container with ID starting with a397efe60f816704df220872baf867090fb9df9737e524bfff5c172d42de33a6 not found: ID does not exist" containerID="a397efe60f816704df220872baf867090fb9df9737e524bfff5c172d42de33a6" Jul 07 16:17:19.116017 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:19.115973 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a397efe60f816704df220872baf867090fb9df9737e524bfff5c172d42de33a6"} err="failed to get container status \"a397efe60f816704df220872baf867090fb9df9737e524bfff5c172d42de33a6\": rpc error: code = NotFound desc = could not find container \"a397efe60f816704df220872baf867090fb9df9737e524bfff5c172d42de33a6\": container with ID starting with a397efe60f816704df220872baf867090fb9df9737e524bfff5c172d42de33a6 not found: ID does not exist" Jul 07 16:17:19.129541 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:19.129519 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-b728d9-openshift-default-6f85f8ff65-gnstx"] Jul 07 16:17:19.134909 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:19.134877 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-b728d9-openshift-default-6f85f8ff65-gnstx"] Jul 07 16:17:19.661356 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:19.661321 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0aae1a92-e503-48d7-8aa3-a04792622460" path="/var/lib/kubelet/pods/0aae1a92-e503-48d7-8aa3-a04792622460/volumes" Jul 07 16:17:21.607138 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:21.607105 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:17:21.957429 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:21.957352 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:17:22.311876 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:22.311844 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-17e556-openshift-default-7dd7c6785f-f4bsw"] Jul 07 16:17:22.312221 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:22.312210 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8fe453f9-345b-4288-8e90-ee666c180de9" containerName="istio-proxy" Jul 07 16:17:22.312262 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:22.312222 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="8fe453f9-345b-4288-8e90-ee666c180de9" containerName="istio-proxy" Jul 07 16:17:22.312262 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:22.312231 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0aae1a92-e503-48d7-8aa3-a04792622460" containerName="istio-proxy" Jul 07 16:17:22.312262 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:22.312237 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="0aae1a92-e503-48d7-8aa3-a04792622460" containerName="istio-proxy" Jul 07 16:17:22.312262 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:22.312246 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0aae1a92-e503-48d7-8aa3-a04792622460" containerName="istio-proxy" Jul 07 16:17:22.312262 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:22.312252 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="0aae1a92-e503-48d7-8aa3-a04792622460" containerName="istio-proxy" Jul 07 16:17:22.312424 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:22.312318 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="8fe453f9-345b-4288-8e90-ee666c180de9" containerName="istio-proxy" Jul 07 16:17:22.312424 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:22.312330 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="0aae1a92-e503-48d7-8aa3-a04792622460" containerName="istio-proxy" Jul 07 16:17:22.312424 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:22.312336 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="0aae1a92-e503-48d7-8aa3-a04792622460" containerName="istio-proxy" Jul 07 16:17:22.314912 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:22.314897 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-17e556-openshift-default-7dd7c6785f-f4bsw" Jul 07 16:17:22.319041 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:22.319021 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-17e556-openshift-default-dockercfg-dqd8g\"" Jul 07 16:17:22.336812 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:22.336789 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-17e556-openshift-default-7dd7c6785f-f4bsw"] Jul 07 16:17:22.338741 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:22.338722 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:17:22.372358 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:22.372309 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f039e558-63f8-49bf-a74e-2d05726d9444-istio-podinfo\") pod \"e2e-shared-b-17e556-openshift-default-7dd7c6785f-f4bsw\" (UID: \"f039e558-63f8-49bf-a74e-2d05726d9444\") " pod="openshift-ingress/e2e-shared-b-17e556-openshift-default-7dd7c6785f-f4bsw" Jul 07 16:17:22.372568 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:22.372362 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f039e558-63f8-49bf-a74e-2d05726d9444-workload-certs\") pod \"e2e-shared-b-17e556-openshift-default-7dd7c6785f-f4bsw\" (UID: \"f039e558-63f8-49bf-a74e-2d05726d9444\") " pod="openshift-ingress/e2e-shared-b-17e556-openshift-default-7dd7c6785f-f4bsw" Jul 07 16:17:22.372568 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:22.372507 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f039e558-63f8-49bf-a74e-2d05726d9444-credential-socket\") pod \"e2e-shared-b-17e556-openshift-default-7dd7c6785f-f4bsw\" (UID: \"f039e558-63f8-49bf-a74e-2d05726d9444\") " pod="openshift-ingress/e2e-shared-b-17e556-openshift-default-7dd7c6785f-f4bsw" Jul 07 16:17:22.372568 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:22.372564 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f039e558-63f8-49bf-a74e-2d05726d9444-istio-envoy\") pod \"e2e-shared-b-17e556-openshift-default-7dd7c6785f-f4bsw\" (UID: \"f039e558-63f8-49bf-a74e-2d05726d9444\") " pod="openshift-ingress/e2e-shared-b-17e556-openshift-default-7dd7c6785f-f4bsw" Jul 07 16:17:22.372745 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:22.372596 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f039e558-63f8-49bf-a74e-2d05726d9444-istio-token\") pod \"e2e-shared-b-17e556-openshift-default-7dd7c6785f-f4bsw\" (UID: \"f039e558-63f8-49bf-a74e-2d05726d9444\") " pod="openshift-ingress/e2e-shared-b-17e556-openshift-default-7dd7c6785f-f4bsw" Jul 07 16:17:22.372745 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:22.372700 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f039e558-63f8-49bf-a74e-2d05726d9444-istiod-ca-cert\") pod \"e2e-shared-b-17e556-openshift-default-7dd7c6785f-f4bsw\" (UID: \"f039e558-63f8-49bf-a74e-2d05726d9444\") " pod="openshift-ingress/e2e-shared-b-17e556-openshift-default-7dd7c6785f-f4bsw" Jul 07 16:17:22.372832 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:22.372776 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52ndv\" (UniqueName: \"kubernetes.io/projected/f039e558-63f8-49bf-a74e-2d05726d9444-kube-api-access-52ndv\") pod \"e2e-shared-b-17e556-openshift-default-7dd7c6785f-f4bsw\" (UID: \"f039e558-63f8-49bf-a74e-2d05726d9444\") " pod="openshift-ingress/e2e-shared-b-17e556-openshift-default-7dd7c6785f-f4bsw" Jul 07 16:17:22.372883 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:22.372838 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f039e558-63f8-49bf-a74e-2d05726d9444-istio-data\") pod \"e2e-shared-b-17e556-openshift-default-7dd7c6785f-f4bsw\" (UID: \"f039e558-63f8-49bf-a74e-2d05726d9444\") " pod="openshift-ingress/e2e-shared-b-17e556-openshift-default-7dd7c6785f-f4bsw" Jul 07 16:17:22.372931 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:22.372883 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f039e558-63f8-49bf-a74e-2d05726d9444-workload-socket\") pod \"e2e-shared-b-17e556-openshift-default-7dd7c6785f-f4bsw\" (UID: \"f039e558-63f8-49bf-a74e-2d05726d9444\") " pod="openshift-ingress/e2e-shared-b-17e556-openshift-default-7dd7c6785f-f4bsw" Jul 07 16:17:22.463266 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:22.463221 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:17:22.474106 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:22.474081 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f039e558-63f8-49bf-a74e-2d05726d9444-istio-envoy\") pod \"e2e-shared-b-17e556-openshift-default-7dd7c6785f-f4bsw\" (UID: \"f039e558-63f8-49bf-a74e-2d05726d9444\") " pod="openshift-ingress/e2e-shared-b-17e556-openshift-default-7dd7c6785f-f4bsw" Jul 07 16:17:22.474221 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:22.474119 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f039e558-63f8-49bf-a74e-2d05726d9444-istio-token\") pod \"e2e-shared-b-17e556-openshift-default-7dd7c6785f-f4bsw\" (UID: \"f039e558-63f8-49bf-a74e-2d05726d9444\") " pod="openshift-ingress/e2e-shared-b-17e556-openshift-default-7dd7c6785f-f4bsw" Jul 07 16:17:22.474221 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:22.474155 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f039e558-63f8-49bf-a74e-2d05726d9444-istiod-ca-cert\") pod \"e2e-shared-b-17e556-openshift-default-7dd7c6785f-f4bsw\" (UID: \"f039e558-63f8-49bf-a74e-2d05726d9444\") " pod="openshift-ingress/e2e-shared-b-17e556-openshift-default-7dd7c6785f-f4bsw" Jul 07 16:17:22.474367 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:22.474341 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52ndv\" (UniqueName: \"kubernetes.io/projected/f039e558-63f8-49bf-a74e-2d05726d9444-kube-api-access-52ndv\") pod \"e2e-shared-b-17e556-openshift-default-7dd7c6785f-f4bsw\" (UID: \"f039e558-63f8-49bf-a74e-2d05726d9444\") " pod="openshift-ingress/e2e-shared-b-17e556-openshift-default-7dd7c6785f-f4bsw" Jul 07 16:17:22.474433 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:22.474417 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f039e558-63f8-49bf-a74e-2d05726d9444-istio-data\") pod \"e2e-shared-b-17e556-openshift-default-7dd7c6785f-f4bsw\" (UID: \"f039e558-63f8-49bf-a74e-2d05726d9444\") " pod="openshift-ingress/e2e-shared-b-17e556-openshift-default-7dd7c6785f-f4bsw" Jul 07 16:17:22.474520 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:22.474468 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f039e558-63f8-49bf-a74e-2d05726d9444-workload-socket\") pod \"e2e-shared-b-17e556-openshift-default-7dd7c6785f-f4bsw\" (UID: \"f039e558-63f8-49bf-a74e-2d05726d9444\") " pod="openshift-ingress/e2e-shared-b-17e556-openshift-default-7dd7c6785f-f4bsw" Jul 07 16:17:22.474596 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:22.474561 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f039e558-63f8-49bf-a74e-2d05726d9444-istio-podinfo\") pod \"e2e-shared-b-17e556-openshift-default-7dd7c6785f-f4bsw\" (UID: \"f039e558-63f8-49bf-a74e-2d05726d9444\") " pod="openshift-ingress/e2e-shared-b-17e556-openshift-default-7dd7c6785f-f4bsw" Jul 07 16:17:22.474596 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:22.474592 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f039e558-63f8-49bf-a74e-2d05726d9444-workload-certs\") pod \"e2e-shared-b-17e556-openshift-default-7dd7c6785f-f4bsw\" (UID: \"f039e558-63f8-49bf-a74e-2d05726d9444\") " pod="openshift-ingress/e2e-shared-b-17e556-openshift-default-7dd7c6785f-f4bsw" Jul 07 16:17:22.474700 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:22.474632 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f039e558-63f8-49bf-a74e-2d05726d9444-credential-socket\") pod \"e2e-shared-b-17e556-openshift-default-7dd7c6785f-f4bsw\" (UID: \"f039e558-63f8-49bf-a74e-2d05726d9444\") " pod="openshift-ingress/e2e-shared-b-17e556-openshift-default-7dd7c6785f-f4bsw" Jul 07 16:17:22.474771 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:22.474744 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f039e558-63f8-49bf-a74e-2d05726d9444-istio-data\") pod \"e2e-shared-b-17e556-openshift-default-7dd7c6785f-f4bsw\" (UID: \"f039e558-63f8-49bf-a74e-2d05726d9444\") " pod="openshift-ingress/e2e-shared-b-17e556-openshift-default-7dd7c6785f-f4bsw" Jul 07 16:17:22.474859 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:22.474837 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f039e558-63f8-49bf-a74e-2d05726d9444-workload-socket\") pod \"e2e-shared-b-17e556-openshift-default-7dd7c6785f-f4bsw\" (UID: \"f039e558-63f8-49bf-a74e-2d05726d9444\") " pod="openshift-ingress/e2e-shared-b-17e556-openshift-default-7dd7c6785f-f4bsw" Jul 07 16:17:22.474956 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:22.474937 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f039e558-63f8-49bf-a74e-2d05726d9444-workload-certs\") pod \"e2e-shared-b-17e556-openshift-default-7dd7c6785f-f4bsw\" (UID: \"f039e558-63f8-49bf-a74e-2d05726d9444\") " pod="openshift-ingress/e2e-shared-b-17e556-openshift-default-7dd7c6785f-f4bsw" Jul 07 16:17:22.475010 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:22.474958 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f039e558-63f8-49bf-a74e-2d05726d9444-credential-socket\") pod \"e2e-shared-b-17e556-openshift-default-7dd7c6785f-f4bsw\" (UID: \"f039e558-63f8-49bf-a74e-2d05726d9444\") " pod="openshift-ingress/e2e-shared-b-17e556-openshift-default-7dd7c6785f-f4bsw" Jul 07 16:17:22.475010 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:22.474944 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f039e558-63f8-49bf-a74e-2d05726d9444-istiod-ca-cert\") pod \"e2e-shared-b-17e556-openshift-default-7dd7c6785f-f4bsw\" (UID: \"f039e558-63f8-49bf-a74e-2d05726d9444\") " pod="openshift-ingress/e2e-shared-b-17e556-openshift-default-7dd7c6785f-f4bsw" Jul 07 16:17:22.476583 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:22.476562 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f039e558-63f8-49bf-a74e-2d05726d9444-istio-envoy\") pod \"e2e-shared-b-17e556-openshift-default-7dd7c6785f-f4bsw\" (UID: \"f039e558-63f8-49bf-a74e-2d05726d9444\") " pod="openshift-ingress/e2e-shared-b-17e556-openshift-default-7dd7c6785f-f4bsw" Jul 07 16:17:22.476971 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:22.476951 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f039e558-63f8-49bf-a74e-2d05726d9444-istio-podinfo\") pod \"e2e-shared-b-17e556-openshift-default-7dd7c6785f-f4bsw\" (UID: \"f039e558-63f8-49bf-a74e-2d05726d9444\") " pod="openshift-ingress/e2e-shared-b-17e556-openshift-default-7dd7c6785f-f4bsw" Jul 07 16:17:22.485154 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:22.485127 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f039e558-63f8-49bf-a74e-2d05726d9444-istio-token\") pod \"e2e-shared-b-17e556-openshift-default-7dd7c6785f-f4bsw\" (UID: \"f039e558-63f8-49bf-a74e-2d05726d9444\") " pod="openshift-ingress/e2e-shared-b-17e556-openshift-default-7dd7c6785f-f4bsw" Jul 07 16:17:22.487930 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:22.487909 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-52ndv\" (UniqueName: \"kubernetes.io/projected/f039e558-63f8-49bf-a74e-2d05726d9444-kube-api-access-52ndv\") pod \"e2e-shared-b-17e556-openshift-default-7dd7c6785f-f4bsw\" (UID: \"f039e558-63f8-49bf-a74e-2d05726d9444\") " pod="openshift-ingress/e2e-shared-b-17e556-openshift-default-7dd7c6785f-f4bsw" Jul 07 16:17:22.625487 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:22.625404 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-17e556-openshift-default-7dd7c6785f-f4bsw" Jul 07 16:17:22.769049 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:22.769016 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-17e556-openshift-default-7dd7c6785f-f4bsw"] Jul 07 16:17:22.770567 ip-10-0-132-65 kubenswrapper[2565]: W0707 16:17:22.770539 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf039e558_63f8_49bf_a74e_2d05726d9444.slice/crio-841f94208770d0cf602657eedd80041cbf6e207a543b5064b3c1118ab7b10444 WatchSource:0}: Error finding container 841f94208770d0cf602657eedd80041cbf6e207a543b5064b3c1118ab7b10444: Status 404 returned error can't find the container with id 841f94208770d0cf602657eedd80041cbf6e207a543b5064b3c1118ab7b10444 Jul 07 16:17:22.772642 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:22.772612 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 07 16:17:22.772706 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:22.772679 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 07 16:17:22.772750 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:22.772705 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 07 16:17:23.118808 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:23.118772 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-17e556-openshift-default-7dd7c6785f-f4bsw" event={"ID":"f039e558-63f8-49bf-a74e-2d05726d9444","Type":"ContainerStarted","Data":"7f44bcc339fd0068c97dc6bbc388cd631f0cc6ee61279de95b49f5786baff1ad"} Jul 07 16:17:23.118995 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:23.118826 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-17e556-openshift-default-7dd7c6785f-f4bsw" event={"ID":"f039e558-63f8-49bf-a74e-2d05726d9444","Type":"ContainerStarted","Data":"841f94208770d0cf602657eedd80041cbf6e207a543b5064b3c1118ab7b10444"} Jul 07 16:17:23.149909 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:23.149854 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-17e556-openshift-default-7dd7c6785f-f4bsw" podStartSLOduration=1.149836352 podStartE2EDuration="1.149836352s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:17:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:17:23.146854591 +0000 UTC m=+2182.103432058" watchObservedRunningTime="2026-07-07 16:17:23.149836352 +0000 UTC m=+2182.106413818" Jul 07 16:17:23.626593 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:23.626555 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-17e556-openshift-default-7dd7c6785f-f4bsw" Jul 07 16:17:23.631682 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:23.631660 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-17e556-openshift-default-7dd7c6785f-f4bsw" Jul 07 16:17:24.123774 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:24.123742 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-17e556-openshift-default-7dd7c6785f-f4bsw" Jul 07 16:17:24.124866 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:24.124845 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-17e556-openshift-default-7dd7c6785f-f4bsw" Jul 07 16:17:25.204510 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:25.204450 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:17:25.344600 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:25.344567 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:17:25.427045 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:25.427016 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:17:26.092754 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:26.092713 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:17:26.193452 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:26.193422 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:17:29.678838 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:29.678804 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:17:30.159038 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:30.159010 2565 generic.go:356] "Generic (PLEG): container finished" podID="0094cc87-2dc6-4827-8288-a3406bf56018" containerID="73f802b80f728d9a02c263972746bca5d3d091b4c8ab630f6f07c491d3abeafe" exitCode=0 Jul 07 16:17:30.159222 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:30.159086 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-382396-openshift-default-66d7949b4c-qq7rq" event={"ID":"0094cc87-2dc6-4827-8288-a3406bf56018","Type":"ContainerDied","Data":"73f802b80f728d9a02c263972746bca5d3d091b4c8ab630f6f07c491d3abeafe"} Jul 07 16:17:30.159520 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:30.159483 2565 scope.go:117] "RemoveContainer" containerID="73f802b80f728d9a02c263972746bca5d3d091b4c8ab630f6f07c491d3abeafe" Jul 07 16:17:30.160796 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:30.160755 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 07 16:17:30.160899 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:30.160844 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 07 16:17:30.160899 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:30.160883 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 07 16:17:31.085516 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:31.081588 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-382396-openshift-default-66d7949b4c-qq7rq" Jul 07 16:17:31.184188 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:31.184146 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-382396-openshift-default-66d7949b4c-qq7rq" event={"ID":"0094cc87-2dc6-4827-8288-a3406bf56018","Type":"ContainerStarted","Data":"23b972ae92f97d0609794dae59eb2b6af58ebc96230ce1133bb4d798df632697"} Jul 07 16:17:31.573813 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:31.573772 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-382396-openshift-default-66d7949b4c-qq7rq" Jul 07 16:17:31.575254 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:31.575223 2565 patch_prober.go:28] interesting pod/e2e-shared-a-382396-openshift-default-66d7949b4c-qq7rq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Jul 07 16:17:31.575506 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:31.575462 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-382396-openshift-default-66d7949b4c-qq7rq" podUID="0094cc87-2dc6-4827-8288-a3406bf56018" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Jul 07 16:17:32.574010 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:32.573960 2565 patch_prober.go:28] interesting pod/e2e-shared-a-382396-openshift-default-66d7949b4c-qq7rq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Jul 07 16:17:32.574539 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:32.574051 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-382396-openshift-default-66d7949b4c-qq7rq" podUID="0094cc87-2dc6-4827-8288-a3406bf56018" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Jul 07 16:17:33.574774 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:33.574265 2565 patch_prober.go:28] interesting pod/e2e-shared-a-382396-openshift-default-66d7949b4c-qq7rq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Jul 07 16:17:33.574774 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:33.574348 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-382396-openshift-default-66d7949b4c-qq7rq" podUID="0094cc87-2dc6-4827-8288-a3406bf56018" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Jul 07 16:17:34.574188 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:34.574148 2565 patch_prober.go:28] interesting pod/e2e-shared-a-382396-openshift-default-66d7949b4c-qq7rq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Jul 07 16:17:34.574537 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:34.574483 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-382396-openshift-default-66d7949b4c-qq7rq" podUID="0094cc87-2dc6-4827-8288-a3406bf56018" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Jul 07 16:17:35.585510 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:35.585457 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-382396-openshift-default-66d7949b4c-qq7rq" Jul 07 16:17:35.585980 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:35.585779 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-382396-openshift-default-66d7949b4c-qq7rq" Jul 07 16:17:35.586660 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:35.586640 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-382396-openshift-default-66d7949b4c-qq7rq" Jul 07 16:17:39.046311 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:39.046271 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:17:39.148385 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:39.148348 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:17:40.165668 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:40.165630 2565 patch_prober.go:28] interesting pod/e2e-shared-b-17e556-openshift-default-7dd7c6785f-f4bsw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 07 16:17:40.167397 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:40.165693 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-17e556-openshift-default-7dd7c6785f-f4bsw" podUID="f039e558-63f8-49bf-a74e-2d05726d9444" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 07 16:17:40.239569 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:40.239536 2565 generic.go:356] "Generic (PLEG): container finished" podID="f039e558-63f8-49bf-a74e-2d05726d9444" containerID="7f44bcc339fd0068c97dc6bbc388cd631f0cc6ee61279de95b49f5786baff1ad" exitCode=0 Jul 07 16:17:40.239729 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:40.239605 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-17e556-openshift-default-7dd7c6785f-f4bsw" event={"ID":"f039e558-63f8-49bf-a74e-2d05726d9444","Type":"ContainerDied","Data":"7f44bcc339fd0068c97dc6bbc388cd631f0cc6ee61279de95b49f5786baff1ad"} Jul 07 16:17:40.240031 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:40.240014 2565 scope.go:117] "RemoveContainer" containerID="7f44bcc339fd0068c97dc6bbc388cd631f0cc6ee61279de95b49f5786baff1ad" Jul 07 16:17:40.241063 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:40.241026 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 07 16:17:40.241138 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:40.241122 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 07 16:17:40.241182 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:40.241162 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 07 16:17:41.247858 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:41.247824 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-17e556-openshift-default-7dd7c6785f-f4bsw" event={"ID":"f039e558-63f8-49bf-a74e-2d05726d9444","Type":"ContainerStarted","Data":"6014ecbce46172a60ad73ab63575beda793cb6b91dc7e5b2865f01515c8ea827"} Jul 07 16:17:41.626574 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:41.626477 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-17e556-openshift-default-7dd7c6785f-f4bsw" Jul 07 16:17:41.626793 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:41.626765 2565 patch_prober.go:28] interesting pod/e2e-shared-b-17e556-openshift-default-7dd7c6785f-f4bsw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Jul 07 16:17:41.626862 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:41.626824 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-17e556-openshift-default-7dd7c6785f-f4bsw" podUID="f039e558-63f8-49bf-a74e-2d05726d9444" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Jul 07 16:17:42.626121 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:42.626081 2565 patch_prober.go:28] interesting pod/e2e-shared-b-17e556-openshift-default-7dd7c6785f-f4bsw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Jul 07 16:17:42.626557 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:42.626149 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-17e556-openshift-default-7dd7c6785f-f4bsw" podUID="f039e558-63f8-49bf-a74e-2d05726d9444" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Jul 07 16:17:43.626127 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:43.626090 2565 patch_prober.go:28] interesting pod/e2e-shared-b-17e556-openshift-default-7dd7c6785f-f4bsw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Jul 07 16:17:43.626588 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:43.626150 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-17e556-openshift-default-7dd7c6785f-f4bsw" podUID="f039e558-63f8-49bf-a74e-2d05726d9444" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Jul 07 16:17:44.626893 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:44.626851 2565 patch_prober.go:28] interesting pod/e2e-shared-b-17e556-openshift-default-7dd7c6785f-f4bsw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Jul 07 16:17:44.627302 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:44.626924 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-17e556-openshift-default-7dd7c6785f-f4bsw" podUID="f039e558-63f8-49bf-a74e-2d05726d9444" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Jul 07 16:17:45.629612 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:45.629582 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-17e556-openshift-default-7dd7c6785f-f4bsw" Jul 07 16:17:45.629999 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:45.629960 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-17e556-openshift-default-7dd7c6785f-f4bsw" Jul 07 16:17:45.630679 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:45.630661 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-17e556-openshift-default-7dd7c6785f-f4bsw" Jul 07 16:17:46.679849 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:46.679818 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:17:47.146982 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:47.146951 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:17:47.281152 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:17:47.281125 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:18:25.533031 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:18:25.532951 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-17e556/auth-test-model-17e556-kserve-6768897c67-nzhs5"] Jul 07 16:18:25.535462 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:18:25.535446 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-17e556/auth-test-model-17e556-kserve-6768897c67-nzhs5" Jul 07 16:18:25.539347 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:18:25.539327 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-17e556\"/\"default-dockercfg-zvrcp\"" Jul 07 16:18:25.539347 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:18:25.539337 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-17e556\"/\"openshift-service-ca.crt\"" Jul 07 16:18:25.539576 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:18:25.539383 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-17e556\"/\"kube-root-ca.crt\"" Jul 07 16:18:25.539576 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:18:25.539526 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-17e556\"/\"auth-test-model-17e556-kserve-self-signed-certs\"" Jul 07 16:18:25.544983 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:18:25.544963 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-17e556/auth-test-model-17e556-kserve-6768897c67-nzhs5"] Jul 07 16:18:25.567881 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:18:25.567852 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:18:25.594654 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:18:25.594624 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5424c9cf-6cba-4a1f-971d-937845eac130-dshm\") pod \"auth-test-model-17e556-kserve-6768897c67-nzhs5\" (UID: \"5424c9cf-6cba-4a1f-971d-937845eac130\") " pod="ai-tenant-e2e-shared-b-17e556/auth-test-model-17e556-kserve-6768897c67-nzhs5" Jul 07 16:18:25.594854 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:18:25.594673 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxztf\" (UniqueName: \"kubernetes.io/projected/5424c9cf-6cba-4a1f-971d-937845eac130-kube-api-access-mxztf\") pod \"auth-test-model-17e556-kserve-6768897c67-nzhs5\" (UID: \"5424c9cf-6cba-4a1f-971d-937845eac130\") " pod="ai-tenant-e2e-shared-b-17e556/auth-test-model-17e556-kserve-6768897c67-nzhs5" Jul 07 16:18:25.594854 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:18:25.594753 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5424c9cf-6cba-4a1f-971d-937845eac130-home\") pod \"auth-test-model-17e556-kserve-6768897c67-nzhs5\" (UID: \"5424c9cf-6cba-4a1f-971d-937845eac130\") " pod="ai-tenant-e2e-shared-b-17e556/auth-test-model-17e556-kserve-6768897c67-nzhs5" Jul 07 16:18:25.594854 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:18:25.594813 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5424c9cf-6cba-4a1f-971d-937845eac130-tmp-dir\") pod \"auth-test-model-17e556-kserve-6768897c67-nzhs5\" (UID: \"5424c9cf-6cba-4a1f-971d-937845eac130\") " pod="ai-tenant-e2e-shared-b-17e556/auth-test-model-17e556-kserve-6768897c67-nzhs5" Jul 07 16:18:25.594991 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:18:25.594862 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5424c9cf-6cba-4a1f-971d-937845eac130-tls-certs\") pod \"auth-test-model-17e556-kserve-6768897c67-nzhs5\" (UID: \"5424c9cf-6cba-4a1f-971d-937845eac130\") " pod="ai-tenant-e2e-shared-b-17e556/auth-test-model-17e556-kserve-6768897c67-nzhs5" Jul 07 16:18:25.594991 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:18:25.594930 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5424c9cf-6cba-4a1f-971d-937845eac130-model-cache\") pod \"auth-test-model-17e556-kserve-6768897c67-nzhs5\" (UID: \"5424c9cf-6cba-4a1f-971d-937845eac130\") " pod="ai-tenant-e2e-shared-b-17e556/auth-test-model-17e556-kserve-6768897c67-nzhs5" Jul 07 16:18:25.695738 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:18:25.695699 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5424c9cf-6cba-4a1f-971d-937845eac130-dshm\") pod \"auth-test-model-17e556-kserve-6768897c67-nzhs5\" (UID: \"5424c9cf-6cba-4a1f-971d-937845eac130\") " pod="ai-tenant-e2e-shared-b-17e556/auth-test-model-17e556-kserve-6768897c67-nzhs5" Jul 07 16:18:25.695952 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:18:25.695765 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxztf\" (UniqueName: \"kubernetes.io/projected/5424c9cf-6cba-4a1f-971d-937845eac130-kube-api-access-mxztf\") pod \"auth-test-model-17e556-kserve-6768897c67-nzhs5\" (UID: \"5424c9cf-6cba-4a1f-971d-937845eac130\") " pod="ai-tenant-e2e-shared-b-17e556/auth-test-model-17e556-kserve-6768897c67-nzhs5" Jul 07 16:18:25.695952 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:18:25.695815 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5424c9cf-6cba-4a1f-971d-937845eac130-home\") pod \"auth-test-model-17e556-kserve-6768897c67-nzhs5\" (UID: \"5424c9cf-6cba-4a1f-971d-937845eac130\") " pod="ai-tenant-e2e-shared-b-17e556/auth-test-model-17e556-kserve-6768897c67-nzhs5" Jul 07 16:18:25.695952 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:18:25.695837 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5424c9cf-6cba-4a1f-971d-937845eac130-tmp-dir\") pod \"auth-test-model-17e556-kserve-6768897c67-nzhs5\" (UID: \"5424c9cf-6cba-4a1f-971d-937845eac130\") " pod="ai-tenant-e2e-shared-b-17e556/auth-test-model-17e556-kserve-6768897c67-nzhs5" Jul 07 16:18:25.695952 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:18:25.695890 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5424c9cf-6cba-4a1f-971d-937845eac130-tls-certs\") pod \"auth-test-model-17e556-kserve-6768897c67-nzhs5\" (UID: \"5424c9cf-6cba-4a1f-971d-937845eac130\") " pod="ai-tenant-e2e-shared-b-17e556/auth-test-model-17e556-kserve-6768897c67-nzhs5" Jul 07 16:18:25.695952 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:18:25.695926 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5424c9cf-6cba-4a1f-971d-937845eac130-model-cache\") pod \"auth-test-model-17e556-kserve-6768897c67-nzhs5\" (UID: \"5424c9cf-6cba-4a1f-971d-937845eac130\") " pod="ai-tenant-e2e-shared-b-17e556/auth-test-model-17e556-kserve-6768897c67-nzhs5" Jul 07 16:18:25.696226 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:18:25.696206 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5424c9cf-6cba-4a1f-971d-937845eac130-home\") pod \"auth-test-model-17e556-kserve-6768897c67-nzhs5\" (UID: \"5424c9cf-6cba-4a1f-971d-937845eac130\") " pod="ai-tenant-e2e-shared-b-17e556/auth-test-model-17e556-kserve-6768897c67-nzhs5" Jul 07 16:18:25.696327 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:18:25.696295 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5424c9cf-6cba-4a1f-971d-937845eac130-model-cache\") pod \"auth-test-model-17e556-kserve-6768897c67-nzhs5\" (UID: \"5424c9cf-6cba-4a1f-971d-937845eac130\") " pod="ai-tenant-e2e-shared-b-17e556/auth-test-model-17e556-kserve-6768897c67-nzhs5" Jul 07 16:18:25.696436 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:18:25.696328 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5424c9cf-6cba-4a1f-971d-937845eac130-tmp-dir\") pod \"auth-test-model-17e556-kserve-6768897c67-nzhs5\" (UID: \"5424c9cf-6cba-4a1f-971d-937845eac130\") " pod="ai-tenant-e2e-shared-b-17e556/auth-test-model-17e556-kserve-6768897c67-nzhs5" Jul 07 16:18:25.698012 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:18:25.697983 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5424c9cf-6cba-4a1f-971d-937845eac130-dshm\") pod \"auth-test-model-17e556-kserve-6768897c67-nzhs5\" (UID: \"5424c9cf-6cba-4a1f-971d-937845eac130\") " pod="ai-tenant-e2e-shared-b-17e556/auth-test-model-17e556-kserve-6768897c67-nzhs5" Jul 07 16:18:25.698329 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:18:25.698312 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5424c9cf-6cba-4a1f-971d-937845eac130-tls-certs\") pod \"auth-test-model-17e556-kserve-6768897c67-nzhs5\" (UID: \"5424c9cf-6cba-4a1f-971d-937845eac130\") " pod="ai-tenant-e2e-shared-b-17e556/auth-test-model-17e556-kserve-6768897c67-nzhs5" Jul 07 16:18:25.705711 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:18:25.705691 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxztf\" (UniqueName: \"kubernetes.io/projected/5424c9cf-6cba-4a1f-971d-937845eac130-kube-api-access-mxztf\") pod \"auth-test-model-17e556-kserve-6768897c67-nzhs5\" (UID: \"5424c9cf-6cba-4a1f-971d-937845eac130\") " pod="ai-tenant-e2e-shared-b-17e556/auth-test-model-17e556-kserve-6768897c67-nzhs5" Jul 07 16:18:25.845872 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:18:25.845837 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-17e556/auth-test-model-17e556-kserve-6768897c67-nzhs5" Jul 07 16:18:25.986642 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:18:25.986617 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-17e556/auth-test-model-17e556-kserve-6768897c67-nzhs5"] Jul 07 16:18:25.989190 ip-10-0-132-65 kubenswrapper[2565]: W0707 16:18:25.989165 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5424c9cf_6cba_4a1f_971d_937845eac130.slice/crio-4ecbcda4fea3e3e7f5758c7d63f06aa61bf0ba0631243a3368e5faca29cab011 WatchSource:0}: Error finding container 4ecbcda4fea3e3e7f5758c7d63f06aa61bf0ba0631243a3368e5faca29cab011: Status 404 returned error can't find the container with id 4ecbcda4fea3e3e7f5758c7d63f06aa61bf0ba0631243a3368e5faca29cab011 Jul 07 16:18:26.471731 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:18:26.471701 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-17e556/auth-test-model-17e556-kserve-6768897c67-nzhs5" event={"ID":"5424c9cf-6cba-4a1f-971d-937845eac130","Type":"ContainerStarted","Data":"dbc890584c1e7f68605d57d0ba481a4c8755eb7c02c91581de315e1ed08266cd"} Jul 07 16:18:26.471731 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:18:26.471739 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-17e556/auth-test-model-17e556-kserve-6768897c67-nzhs5" event={"ID":"5424c9cf-6cba-4a1f-971d-937845eac130","Type":"ContainerStarted","Data":"4ecbcda4fea3e3e7f5758c7d63f06aa61bf0ba0631243a3368e5faca29cab011"} Jul 07 16:18:26.497011 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:18:26.496959 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-17e556/auth-test-model-17e556-kserve-6768897c67-nzhs5" podStartSLOduration=1.496945585 podStartE2EDuration="1.496945585s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:18:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:18:26.495034036 +0000 UTC m=+2245.451611504" watchObservedRunningTime="2026-07-07 16:18:26.496945585 +0000 UTC m=+2245.453523081" Jul 07 16:18:35.845989 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:18:35.845953 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-17e556/auth-test-model-17e556-kserve-6768897c67-nzhs5" Jul 07 16:18:35.845989 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:18:35.846008 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-17e556/auth-test-model-17e556-kserve-6768897c67-nzhs5" Jul 07 16:18:35.858671 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:18:35.858640 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-17e556/auth-test-model-17e556-kserve-6768897c67-nzhs5" Jul 07 16:18:36.540230 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:18:36.540202 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-17e556/auth-test-model-17e556-kserve-6768897c67-nzhs5" Jul 07 16:19:29.875103 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:19:29.875064 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:20:28.877171 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:20:28.877112 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:20:29.734038 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:20:29.733999 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-17e556/auth-test-model-17e556-kserve-6768897c67-nzhs5"] Jul 07 16:20:29.754351 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:20:29.754294 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:20:37.947624 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:20:37.947575 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:20:44.754399 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:20:44.754361 2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-17e556/auth-test-model-17e556-kserve-6768897c67-nzhs5" podUID="5424c9cf-6cba-4a1f-971d-937845eac130" containerName="main" containerID="cri-o://dbc890584c1e7f68605d57d0ba481a4c8755eb7c02c91581de315e1ed08266cd" gracePeriod=45 Jul 07 16:20:45.003244 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:20:45.003217 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-17e556/auth-test-model-17e556-kserve-6768897c67-nzhs5" Jul 07 16:20:45.130190 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:20:45.130159 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5424c9cf-6cba-4a1f-971d-937845eac130-dshm\") pod \"5424c9cf-6cba-4a1f-971d-937845eac130\" (UID: \"5424c9cf-6cba-4a1f-971d-937845eac130\") " Jul 07 16:20:45.130375 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:20:45.130207 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5424c9cf-6cba-4a1f-971d-937845eac130-home\") pod \"5424c9cf-6cba-4a1f-971d-937845eac130\" (UID: \"5424c9cf-6cba-4a1f-971d-937845eac130\") " Jul 07 16:20:45.130375 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:20:45.130330 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5424c9cf-6cba-4a1f-971d-937845eac130-tmp-dir\") pod \"5424c9cf-6cba-4a1f-971d-937845eac130\" (UID: \"5424c9cf-6cba-4a1f-971d-937845eac130\") " Jul 07 16:20:45.130448 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:20:45.130425 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5424c9cf-6cba-4a1f-971d-937845eac130-tls-certs\") pod \"5424c9cf-6cba-4a1f-971d-937845eac130\" (UID: \"5424c9cf-6cba-4a1f-971d-937845eac130\") " Jul 07 16:20:45.130487 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:20:45.130466 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5424c9cf-6cba-4a1f-971d-937845eac130-model-cache\") pod \"5424c9cf-6cba-4a1f-971d-937845eac130\" (UID: \"5424c9cf-6cba-4a1f-971d-937845eac130\") " Jul 07 16:20:45.130487 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:20:45.130465 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5424c9cf-6cba-4a1f-971d-937845eac130-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5424c9cf-6cba-4a1f-971d-937845eac130" (UID: "5424c9cf-6cba-4a1f-971d-937845eac130"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 16:20:45.130607 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:20:45.130536 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5424c9cf-6cba-4a1f-971d-937845eac130-home" (OuterVolumeSpecName: "home") pod "5424c9cf-6cba-4a1f-971d-937845eac130" (UID: "5424c9cf-6cba-4a1f-971d-937845eac130"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 16:20:45.130607 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:20:45.130543 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mxztf\" (UniqueName: \"kubernetes.io/projected/5424c9cf-6cba-4a1f-971d-937845eac130-kube-api-access-mxztf\") pod \"5424c9cf-6cba-4a1f-971d-937845eac130\" (UID: \"5424c9cf-6cba-4a1f-971d-937845eac130\") " Jul 07 16:20:45.130726 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:20:45.130696 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5424c9cf-6cba-4a1f-971d-937845eac130-model-cache" (OuterVolumeSpecName: "model-cache") pod "5424c9cf-6cba-4a1f-971d-937845eac130" (UID: "5424c9cf-6cba-4a1f-971d-937845eac130"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 16:20:45.130858 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:20:45.130843 2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5424c9cf-6cba-4a1f-971d-937845eac130-home\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 07 16:20:45.130858 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:20:45.130858 2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5424c9cf-6cba-4a1f-971d-937845eac130-tmp-dir\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 07 16:20:45.130950 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:20:45.130867 2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5424c9cf-6cba-4a1f-971d-937845eac130-model-cache\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 07 16:20:45.132516 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:20:45.132462 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5424c9cf-6cba-4a1f-971d-937845eac130-dshm" (OuterVolumeSpecName: "dshm") pod "5424c9cf-6cba-4a1f-971d-937845eac130" (UID: "5424c9cf-6cba-4a1f-971d-937845eac130"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 16:20:45.132633 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:20:45.132514 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5424c9cf-6cba-4a1f-971d-937845eac130-kube-api-access-mxztf" (OuterVolumeSpecName: "kube-api-access-mxztf") pod "5424c9cf-6cba-4a1f-971d-937845eac130" (UID: "5424c9cf-6cba-4a1f-971d-937845eac130"). InnerVolumeSpecName "kube-api-access-mxztf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 16:20:45.132633 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:20:45.132558 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5424c9cf-6cba-4a1f-971d-937845eac130-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5424c9cf-6cba-4a1f-971d-937845eac130" (UID: "5424c9cf-6cba-4a1f-971d-937845eac130"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 16:20:45.166764 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:20:45.166730 2565 generic.go:356] "Generic (PLEG): container finished" podID="5424c9cf-6cba-4a1f-971d-937845eac130" containerID="dbc890584c1e7f68605d57d0ba481a4c8755eb7c02c91581de315e1ed08266cd" exitCode=0 Jul 07 16:20:45.166764 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:20:45.166765 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-17e556/auth-test-model-17e556-kserve-6768897c67-nzhs5" Jul 07 16:20:45.166948 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:20:45.166802 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-17e556/auth-test-model-17e556-kserve-6768897c67-nzhs5" event={"ID":"5424c9cf-6cba-4a1f-971d-937845eac130","Type":"ContainerDied","Data":"dbc890584c1e7f68605d57d0ba481a4c8755eb7c02c91581de315e1ed08266cd"} Jul 07 16:20:45.166948 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:20:45.166846 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-17e556/auth-test-model-17e556-kserve-6768897c67-nzhs5" event={"ID":"5424c9cf-6cba-4a1f-971d-937845eac130","Type":"ContainerDied","Data":"4ecbcda4fea3e3e7f5758c7d63f06aa61bf0ba0631243a3368e5faca29cab011"} Jul 07 16:20:45.166948 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:20:45.166863 2565 scope.go:117] "RemoveContainer" containerID="dbc890584c1e7f68605d57d0ba481a4c8755eb7c02c91581de315e1ed08266cd" Jul 07 16:20:45.175960 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:20:45.175939 2565 scope.go:117] "RemoveContainer" containerID="dbc890584c1e7f68605d57d0ba481a4c8755eb7c02c91581de315e1ed08266cd" Jul 07 16:20:45.176192 ip-10-0-132-65 kubenswrapper[2565]: E0707 16:20:45.176175 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dbc890584c1e7f68605d57d0ba481a4c8755eb7c02c91581de315e1ed08266cd\": container with ID starting with dbc890584c1e7f68605d57d0ba481a4c8755eb7c02c91581de315e1ed08266cd not found: ID does not exist" containerID="dbc890584c1e7f68605d57d0ba481a4c8755eb7c02c91581de315e1ed08266cd" Jul 07 16:20:45.176261 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:20:45.176199 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dbc890584c1e7f68605d57d0ba481a4c8755eb7c02c91581de315e1ed08266cd"} err="failed to get container status \"dbc890584c1e7f68605d57d0ba481a4c8755eb7c02c91581de315e1ed08266cd\": rpc error: code = NotFound desc = could not find container \"dbc890584c1e7f68605d57d0ba481a4c8755eb7c02c91581de315e1ed08266cd\": container with ID starting with dbc890584c1e7f68605d57d0ba481a4c8755eb7c02c91581de315e1ed08266cd not found: ID does not exist" Jul 07 16:20:45.192108 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:20:45.192074 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-17e556/auth-test-model-17e556-kserve-6768897c67-nzhs5"] Jul 07 16:20:45.196479 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:20:45.196457 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-17e556/auth-test-model-17e556-kserve-6768897c67-nzhs5"] Jul 07 16:20:45.231398 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:20:45.231377 2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5424c9cf-6cba-4a1f-971d-937845eac130-tls-certs\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 07 16:20:45.231398 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:20:45.231398 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mxztf\" (UniqueName: \"kubernetes.io/projected/5424c9cf-6cba-4a1f-971d-937845eac130-kube-api-access-mxztf\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 07 16:20:45.231570 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:20:45.231407 2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5424c9cf-6cba-4a1f-971d-937845eac130-dshm\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 07 16:20:45.661693 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:20:45.661659 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5424c9cf-6cba-4a1f-971d-937845eac130" path="/var/lib/kubelet/pods/5424c9cf-6cba-4a1f-971d-937845eac130/volumes" Jul 07 16:20:54.668262 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:20:54.668215 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:20:54.766094 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:20:54.766052 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:21:01.471639 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:21:01.471606 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-17e556/sub-test-model-17e556-kserve-6c7867756f-dwms7"] Jul 07 16:21:01.472164 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:21:01.472012 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5424c9cf-6cba-4a1f-971d-937845eac130" containerName="main" Jul 07 16:21:01.472164 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:21:01.472027 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="5424c9cf-6cba-4a1f-971d-937845eac130" containerName="main" Jul 07 16:21:01.472164 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:21:01.472122 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="5424c9cf-6cba-4a1f-971d-937845eac130" containerName="main" Jul 07 16:21:01.475262 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:21:01.475236 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-17e556/sub-test-model-17e556-kserve-6c7867756f-dwms7" Jul 07 16:21:01.479453 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:21:01.479431 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-17e556\"/\"kube-root-ca.crt\"" Jul 07 16:21:01.480941 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:21:01.480918 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-17e556\"/\"sub-test-model-17e556-kserve-self-signed-certs\"" Jul 07 16:21:01.481266 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:21:01.480934 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-17e556\"/\"default-dockercfg-zvrcp\"" Jul 07 16:21:01.481414 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:21:01.481003 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-17e556\"/\"openshift-service-ca.crt\"" Jul 07 16:21:01.483228 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:21:01.483188 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tlxjb\" (UniqueName: \"kubernetes.io/projected/96a93e1b-b1cb-4f98-bf0e-f134045ea199-kube-api-access-tlxjb\") pod \"sub-test-model-17e556-kserve-6c7867756f-dwms7\" (UID: \"96a93e1b-b1cb-4f98-bf0e-f134045ea199\") " pod="ai-tenant-e2e-shared-b-17e556/sub-test-model-17e556-kserve-6c7867756f-dwms7" Jul 07 16:21:01.483331 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:21:01.483272 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/96a93e1b-b1cb-4f98-bf0e-f134045ea199-home\") pod \"sub-test-model-17e556-kserve-6c7867756f-dwms7\" (UID: \"96a93e1b-b1cb-4f98-bf0e-f134045ea199\") " pod="ai-tenant-e2e-shared-b-17e556/sub-test-model-17e556-kserve-6c7867756f-dwms7" Jul 07 16:21:01.483331 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:21:01.483315 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/96a93e1b-b1cb-4f98-bf0e-f134045ea199-dshm\") pod \"sub-test-model-17e556-kserve-6c7867756f-dwms7\" (UID: \"96a93e1b-b1cb-4f98-bf0e-f134045ea199\") " pod="ai-tenant-e2e-shared-b-17e556/sub-test-model-17e556-kserve-6c7867756f-dwms7" Jul 07 16:21:01.483447 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:21:01.483408 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96a93e1b-b1cb-4f98-bf0e-f134045ea199-tmp-dir\") pod \"sub-test-model-17e556-kserve-6c7867756f-dwms7\" (UID: \"96a93e1b-b1cb-4f98-bf0e-f134045ea199\") " pod="ai-tenant-e2e-shared-b-17e556/sub-test-model-17e556-kserve-6c7867756f-dwms7" Jul 07 16:21:01.483531 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:21:01.483474 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/96a93e1b-b1cb-4f98-bf0e-f134045ea199-model-cache\") pod \"sub-test-model-17e556-kserve-6c7867756f-dwms7\" (UID: \"96a93e1b-b1cb-4f98-bf0e-f134045ea199\") " pod="ai-tenant-e2e-shared-b-17e556/sub-test-model-17e556-kserve-6c7867756f-dwms7" Jul 07 16:21:01.483599 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:21:01.483572 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/96a93e1b-b1cb-4f98-bf0e-f134045ea199-tls-certs\") pod \"sub-test-model-17e556-kserve-6c7867756f-dwms7\" (UID: \"96a93e1b-b1cb-4f98-bf0e-f134045ea199\") " pod="ai-tenant-e2e-shared-b-17e556/sub-test-model-17e556-kserve-6c7867756f-dwms7" Jul 07 16:21:01.492193 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:21:01.492173 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-17e556/sub-test-model-17e556-kserve-6c7867756f-dwms7"] Jul 07 16:21:01.523552 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:21:01.523520 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:21:01.584408 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:21:01.584372 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tlxjb\" (UniqueName: \"kubernetes.io/projected/96a93e1b-b1cb-4f98-bf0e-f134045ea199-kube-api-access-tlxjb\") pod \"sub-test-model-17e556-kserve-6c7867756f-dwms7\" (UID: \"96a93e1b-b1cb-4f98-bf0e-f134045ea199\") " pod="ai-tenant-e2e-shared-b-17e556/sub-test-model-17e556-kserve-6c7867756f-dwms7" Jul 07 16:21:01.584621 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:21:01.584440 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/96a93e1b-b1cb-4f98-bf0e-f134045ea199-home\") pod \"sub-test-model-17e556-kserve-6c7867756f-dwms7\" (UID: \"96a93e1b-b1cb-4f98-bf0e-f134045ea199\") " pod="ai-tenant-e2e-shared-b-17e556/sub-test-model-17e556-kserve-6c7867756f-dwms7" Jul 07 16:21:01.584621 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:21:01.584465 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/96a93e1b-b1cb-4f98-bf0e-f134045ea199-dshm\") pod \"sub-test-model-17e556-kserve-6c7867756f-dwms7\" (UID: \"96a93e1b-b1cb-4f98-bf0e-f134045ea199\") " pod="ai-tenant-e2e-shared-b-17e556/sub-test-model-17e556-kserve-6c7867756f-dwms7" Jul 07 16:21:01.584621 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:21:01.584540 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96a93e1b-b1cb-4f98-bf0e-f134045ea199-tmp-dir\") pod \"sub-test-model-17e556-kserve-6c7867756f-dwms7\" (UID: \"96a93e1b-b1cb-4f98-bf0e-f134045ea199\") " pod="ai-tenant-e2e-shared-b-17e556/sub-test-model-17e556-kserve-6c7867756f-dwms7" Jul 07 16:21:01.584621 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:21:01.584594 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/96a93e1b-b1cb-4f98-bf0e-f134045ea199-model-cache\") pod \"sub-test-model-17e556-kserve-6c7867756f-dwms7\" (UID: \"96a93e1b-b1cb-4f98-bf0e-f134045ea199\") " pod="ai-tenant-e2e-shared-b-17e556/sub-test-model-17e556-kserve-6c7867756f-dwms7" Jul 07 16:21:01.584621 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:21:01.584617 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/96a93e1b-b1cb-4f98-bf0e-f134045ea199-tls-certs\") pod \"sub-test-model-17e556-kserve-6c7867756f-dwms7\" (UID: \"96a93e1b-b1cb-4f98-bf0e-f134045ea199\") " pod="ai-tenant-e2e-shared-b-17e556/sub-test-model-17e556-kserve-6c7867756f-dwms7" Jul 07 16:21:01.584902 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:21:01.584875 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/96a93e1b-b1cb-4f98-bf0e-f134045ea199-home\") pod \"sub-test-model-17e556-kserve-6c7867756f-dwms7\" (UID: \"96a93e1b-b1cb-4f98-bf0e-f134045ea199\") " pod="ai-tenant-e2e-shared-b-17e556/sub-test-model-17e556-kserve-6c7867756f-dwms7" Jul 07 16:21:01.584970 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:21:01.584936 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96a93e1b-b1cb-4f98-bf0e-f134045ea199-tmp-dir\") pod \"sub-test-model-17e556-kserve-6c7867756f-dwms7\" (UID: \"96a93e1b-b1cb-4f98-bf0e-f134045ea199\") " pod="ai-tenant-e2e-shared-b-17e556/sub-test-model-17e556-kserve-6c7867756f-dwms7" Jul 07 16:21:01.585095 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:21:01.585074 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/96a93e1b-b1cb-4f98-bf0e-f134045ea199-model-cache\") pod \"sub-test-model-17e556-kserve-6c7867756f-dwms7\" (UID: \"96a93e1b-b1cb-4f98-bf0e-f134045ea199\") " pod="ai-tenant-e2e-shared-b-17e556/sub-test-model-17e556-kserve-6c7867756f-dwms7" Jul 07 16:21:01.586759 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:21:01.586737 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/96a93e1b-b1cb-4f98-bf0e-f134045ea199-dshm\") pod \"sub-test-model-17e556-kserve-6c7867756f-dwms7\" (UID: \"96a93e1b-b1cb-4f98-bf0e-f134045ea199\") " pod="ai-tenant-e2e-shared-b-17e556/sub-test-model-17e556-kserve-6c7867756f-dwms7" Jul 07 16:21:01.587087 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:21:01.587065 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/96a93e1b-b1cb-4f98-bf0e-f134045ea199-tls-certs\") pod \"sub-test-model-17e556-kserve-6c7867756f-dwms7\" (UID: \"96a93e1b-b1cb-4f98-bf0e-f134045ea199\") " pod="ai-tenant-e2e-shared-b-17e556/sub-test-model-17e556-kserve-6c7867756f-dwms7" Jul 07 16:21:01.602209 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:21:01.602188 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tlxjb\" (UniqueName: \"kubernetes.io/projected/96a93e1b-b1cb-4f98-bf0e-f134045ea199-kube-api-access-tlxjb\") pod \"sub-test-model-17e556-kserve-6c7867756f-dwms7\" (UID: \"96a93e1b-b1cb-4f98-bf0e-f134045ea199\") " pod="ai-tenant-e2e-shared-b-17e556/sub-test-model-17e556-kserve-6c7867756f-dwms7" Jul 07 16:21:01.694442 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:21:01.694402 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-s9kl8_62da0165-d32c-4bba-9844-a49154d765e5/console-operator/1.log" Jul 07 16:21:01.698748 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:21:01.698729 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-942xh_31ff16f0-6054-4eba-942a-b07a5a7afac7/ovn-acl-logging/0.log" Jul 07 16:21:01.699323 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:21:01.699306 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-s9kl8_62da0165-d32c-4bba-9844-a49154d765e5/console-operator/1.log" Jul 07 16:21:01.703729 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:21:01.703711 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-942xh_31ff16f0-6054-4eba-942a-b07a5a7afac7/ovn-acl-logging/0.log" Jul 07 16:21:01.787943 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:21:01.787862 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-17e556/sub-test-model-17e556-kserve-6c7867756f-dwms7" Jul 07 16:21:01.930539 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:21:01.930478 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-17e556/sub-test-model-17e556-kserve-6c7867756f-dwms7"] Jul 07 16:21:01.933844 ip-10-0-132-65 kubenswrapper[2565]: W0707 16:21:01.933819 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod96a93e1b_b1cb_4f98_bf0e_f134045ea199.slice/crio-ab9dba25eab5ba6e49cc5a438395d91c254b181d0c8fa55adaa5912e2990606b WatchSource:0}: Error finding container ab9dba25eab5ba6e49cc5a438395d91c254b181d0c8fa55adaa5912e2990606b: Status 404 returned error can't find the container with id ab9dba25eab5ba6e49cc5a438395d91c254b181d0c8fa55adaa5912e2990606b Jul 07 16:21:02.253589 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:21:02.253547 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-17e556/sub-test-model-17e556-kserve-6c7867756f-dwms7" event={"ID":"96a93e1b-b1cb-4f98-bf0e-f134045ea199","Type":"ContainerStarted","Data":"d2b081f23c9a16009891680753ad2bb5634737629b40a29af2d6253b3bbad19b"} Jul 07 16:21:02.253589 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:21:02.253594 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-17e556/sub-test-model-17e556-kserve-6c7867756f-dwms7" event={"ID":"96a93e1b-b1cb-4f98-bf0e-f134045ea199","Type":"ContainerStarted","Data":"ab9dba25eab5ba6e49cc5a438395d91c254b181d0c8fa55adaa5912e2990606b"} Jul 07 16:21:02.278913 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:21:02.278870 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-17e556/sub-test-model-17e556-kserve-6c7867756f-dwms7" podStartSLOduration=1.278856283 podStartE2EDuration="1.278856283s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:21:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:21:02.276715693 +0000 UTC m=+2401.233293152" watchObservedRunningTime="2026-07-07 16:21:02.278856283 +0000 UTC m=+2401.235433750" Jul 07 16:21:11.788689 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:21:11.788632 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-17e556/sub-test-model-17e556-kserve-6c7867756f-dwms7" Jul 07 16:21:11.788689 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:21:11.788689 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-17e556/sub-test-model-17e556-kserve-6c7867756f-dwms7" Jul 07 16:21:11.801367 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:21:11.801342 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-17e556/sub-test-model-17e556-kserve-6c7867756f-dwms7" Jul 07 16:21:12.320400 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:21:12.320369 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-17e556/sub-test-model-17e556-kserve-6c7867756f-dwms7" Jul 07 16:21:47.262940 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:21:47.262903 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:21:48.144553 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:21:48.144473 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-17e556/sub-test-model-17e556-kserve-6c7867756f-dwms7"] Jul 07 16:21:48.171169 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:21:48.171137 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:21:53.823696 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:21:53.823662 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:22:03.166620 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:22:03.166576 2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-17e556/sub-test-model-17e556-kserve-6c7867756f-dwms7" podUID="96a93e1b-b1cb-4f98-bf0e-f134045ea199" containerName="main" containerID="cri-o://d2b081f23c9a16009891680753ad2bb5634737629b40a29af2d6253b3bbad19b" gracePeriod=45 Jul 07 16:22:03.419949 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:22:03.419887 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-17e556/sub-test-model-17e556-kserve-6c7867756f-dwms7" Jul 07 16:22:03.500340 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:22:03.500306 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tlxjb\" (UniqueName: \"kubernetes.io/projected/96a93e1b-b1cb-4f98-bf0e-f134045ea199-kube-api-access-tlxjb\") pod \"96a93e1b-b1cb-4f98-bf0e-f134045ea199\" (UID: \"96a93e1b-b1cb-4f98-bf0e-f134045ea199\") " Jul 07 16:22:03.500571 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:22:03.500377 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/96a93e1b-b1cb-4f98-bf0e-f134045ea199-model-cache\") pod \"96a93e1b-b1cb-4f98-bf0e-f134045ea199\" (UID: \"96a93e1b-b1cb-4f98-bf0e-f134045ea199\") " Jul 07 16:22:03.500571 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:22:03.500477 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/96a93e1b-b1cb-4f98-bf0e-f134045ea199-tls-certs\") pod \"96a93e1b-b1cb-4f98-bf0e-f134045ea199\" (UID: \"96a93e1b-b1cb-4f98-bf0e-f134045ea199\") " Jul 07 16:22:03.500706 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:22:03.500580 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96a93e1b-b1cb-4f98-bf0e-f134045ea199-tmp-dir\") pod \"96a93e1b-b1cb-4f98-bf0e-f134045ea199\" (UID: \"96a93e1b-b1cb-4f98-bf0e-f134045ea199\") " Jul 07 16:22:03.500706 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:22:03.500610 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/96a93e1b-b1cb-4f98-bf0e-f134045ea199-dshm\") pod \"96a93e1b-b1cb-4f98-bf0e-f134045ea199\" (UID: \"96a93e1b-b1cb-4f98-bf0e-f134045ea199\") " Jul 07 16:22:03.500706 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:22:03.500671 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/96a93e1b-b1cb-4f98-bf0e-f134045ea199-home\") pod \"96a93e1b-b1cb-4f98-bf0e-f134045ea199\" (UID: \"96a93e1b-b1cb-4f98-bf0e-f134045ea199\") " Jul 07 16:22:03.500706 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:22:03.500678 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96a93e1b-b1cb-4f98-bf0e-f134045ea199-model-cache" (OuterVolumeSpecName: "model-cache") pod "96a93e1b-b1cb-4f98-bf0e-f134045ea199" (UID: "96a93e1b-b1cb-4f98-bf0e-f134045ea199"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 16:22:03.500923 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:22:03.500876 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96a93e1b-b1cb-4f98-bf0e-f134045ea199-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "96a93e1b-b1cb-4f98-bf0e-f134045ea199" (UID: "96a93e1b-b1cb-4f98-bf0e-f134045ea199"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 16:22:03.501020 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:22:03.500992 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96a93e1b-b1cb-4f98-bf0e-f134045ea199-home" (OuterVolumeSpecName: "home") pod "96a93e1b-b1cb-4f98-bf0e-f134045ea199" (UID: "96a93e1b-b1cb-4f98-bf0e-f134045ea199"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 16:22:03.501087 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:22:03.501004 2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96a93e1b-b1cb-4f98-bf0e-f134045ea199-tmp-dir\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 07 16:22:03.501087 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:22:03.501048 2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/96a93e1b-b1cb-4f98-bf0e-f134045ea199-model-cache\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 07 16:22:03.502560 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:22:03.502526 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/96a93e1b-b1cb-4f98-bf0e-f134045ea199-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "96a93e1b-b1cb-4f98-bf0e-f134045ea199" (UID: "96a93e1b-b1cb-4f98-bf0e-f134045ea199"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 16:22:03.502670 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:22:03.502612 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96a93e1b-b1cb-4f98-bf0e-f134045ea199-dshm" (OuterVolumeSpecName: "dshm") pod "96a93e1b-b1cb-4f98-bf0e-f134045ea199" (UID: "96a93e1b-b1cb-4f98-bf0e-f134045ea199"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 16:22:03.502715 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:22:03.502668 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96a93e1b-b1cb-4f98-bf0e-f134045ea199-kube-api-access-tlxjb" (OuterVolumeSpecName: "kube-api-access-tlxjb") pod "96a93e1b-b1cb-4f98-bf0e-f134045ea199" (UID: "96a93e1b-b1cb-4f98-bf0e-f134045ea199"). InnerVolumeSpecName "kube-api-access-tlxjb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 16:22:03.566458 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:22:03.566426 2565 generic.go:356] "Generic (PLEG): container finished" podID="96a93e1b-b1cb-4f98-bf0e-f134045ea199" containerID="d2b081f23c9a16009891680753ad2bb5634737629b40a29af2d6253b3bbad19b" exitCode=0 Jul 07 16:22:03.566648 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:22:03.566481 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-17e556/sub-test-model-17e556-kserve-6c7867756f-dwms7" Jul 07 16:22:03.566648 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:22:03.566523 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-17e556/sub-test-model-17e556-kserve-6c7867756f-dwms7" event={"ID":"96a93e1b-b1cb-4f98-bf0e-f134045ea199","Type":"ContainerDied","Data":"d2b081f23c9a16009891680753ad2bb5634737629b40a29af2d6253b3bbad19b"} Jul 07 16:22:03.566648 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:22:03.566567 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-17e556/sub-test-model-17e556-kserve-6c7867756f-dwms7" event={"ID":"96a93e1b-b1cb-4f98-bf0e-f134045ea199","Type":"ContainerDied","Data":"ab9dba25eab5ba6e49cc5a438395d91c254b181d0c8fa55adaa5912e2990606b"} Jul 07 16:22:03.566648 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:22:03.566588 2565 scope.go:117] "RemoveContainer" containerID="d2b081f23c9a16009891680753ad2bb5634737629b40a29af2d6253b3bbad19b" Jul 07 16:22:03.578378 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:22:03.578358 2565 scope.go:117] "RemoveContainer" containerID="d2b081f23c9a16009891680753ad2bb5634737629b40a29af2d6253b3bbad19b" Jul 07 16:22:03.578703 ip-10-0-132-65 kubenswrapper[2565]: E0707 16:22:03.578685 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2b081f23c9a16009891680753ad2bb5634737629b40a29af2d6253b3bbad19b\": container with ID starting with d2b081f23c9a16009891680753ad2bb5634737629b40a29af2d6253b3bbad19b not found: ID does not exist" containerID="d2b081f23c9a16009891680753ad2bb5634737629b40a29af2d6253b3bbad19b" Jul 07 16:22:03.578758 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:22:03.578736 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2b081f23c9a16009891680753ad2bb5634737629b40a29af2d6253b3bbad19b"} err="failed to get container status \"d2b081f23c9a16009891680753ad2bb5634737629b40a29af2d6253b3bbad19b\": rpc error: code = NotFound desc = could not find container \"d2b081f23c9a16009891680753ad2bb5634737629b40a29af2d6253b3bbad19b\": container with ID starting with d2b081f23c9a16009891680753ad2bb5634737629b40a29af2d6253b3bbad19b not found: ID does not exist" Jul 07 16:22:03.591893 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:22:03.591864 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-17e556/sub-test-model-17e556-kserve-6c7867756f-dwms7"] Jul 07 16:22:03.596399 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:22:03.596377 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-17e556/sub-test-model-17e556-kserve-6c7867756f-dwms7"] Jul 07 16:22:03.601752 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:22:03.601732 2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/96a93e1b-b1cb-4f98-bf0e-f134045ea199-tls-certs\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 07 16:22:03.601818 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:22:03.601756 2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/96a93e1b-b1cb-4f98-bf0e-f134045ea199-dshm\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 07 16:22:03.601818 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:22:03.601769 2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/96a93e1b-b1cb-4f98-bf0e-f134045ea199-home\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 07 16:22:03.601818 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:22:03.601778 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tlxjb\" (UniqueName: \"kubernetes.io/projected/96a93e1b-b1cb-4f98-bf0e-f134045ea199-kube-api-access-tlxjb\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 07 16:22:03.661650 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:22:03.661619 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="96a93e1b-b1cb-4f98-bf0e-f134045ea199" path="/var/lib/kubelet/pods/96a93e1b-b1cb-4f98-bf0e-f134045ea199/volumes" Jul 07 16:22:15.328766 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:22:15.328732 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:22:16.017352 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:22:16.017320 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:22:39.903549 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:22:39.903512 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:22:40.004815 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:22:40.004773 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:22:48.097343 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:22:48.097312 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:22:48.199089 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:22:48.199047 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:22:57.885630 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:22:57.885548 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:23:15.479151 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:23:15.479121 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:23:15.555544 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:23:15.555485 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:23:22.945234 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:23:22.945202 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:23:23.763548 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:23:23.763481 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:23:24.883813 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:23:24.883779 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:23:25.283442 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:23:25.283361 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:23:25.383108 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:23:25.383070 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:23:25.484099 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:23:25.484047 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:23:25.683549 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:23:25.683514 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:23:30.483320 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:23:30.483286 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:24:08.196649 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:24:08.196614 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:24:08.462389 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:24:08.462289 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-382396-openshift-default-66d7949b4c-qq7rq"] Jul 07 16:24:08.462579 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:24:08.462553 2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-382396-openshift-default-66d7949b4c-qq7rq" podUID="0094cc87-2dc6-4827-8288-a3406bf56018" containerName="istio-proxy" containerID="cri-o://23b972ae92f97d0609794dae59eb2b6af58ebc96230ce1133bb4d798df632697" gracePeriod=30 Jul 07 16:24:08.467706 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:24:08.467678 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:24:08.724139 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:24:08.724047 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:24:08.896903 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:24:08.896872 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:24:09.295307 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:24:09.295268 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:24:09.460228 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:24:09.460197 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:24:09.560404 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:24:09.560376 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:24:09.762805 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:24:09.762777 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:24:13.720554 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:24:13.720528 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-382396-openshift-default-66d7949b4c-qq7rq" Jul 07 16:24:13.810382 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:24:13.810350 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0094cc87-2dc6-4827-8288-a3406bf56018-istio-envoy\") pod \"0094cc87-2dc6-4827-8288-a3406bf56018\" (UID: \"0094cc87-2dc6-4827-8288-a3406bf56018\") " Jul 07 16:24:13.810597 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:24:13.810415 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0094cc87-2dc6-4827-8288-a3406bf56018-istio-token\") pod \"0094cc87-2dc6-4827-8288-a3406bf56018\" (UID: \"0094cc87-2dc6-4827-8288-a3406bf56018\") " Jul 07 16:24:13.810597 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:24:13.810553 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0094cc87-2dc6-4827-8288-a3406bf56018-istiod-ca-cert\") pod \"0094cc87-2dc6-4827-8288-a3406bf56018\" (UID: \"0094cc87-2dc6-4827-8288-a3406bf56018\") " Jul 07 16:24:13.810597 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:24:13.810588 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0094cc87-2dc6-4827-8288-a3406bf56018-workload-socket\") pod \"0094cc87-2dc6-4827-8288-a3406bf56018\" (UID: \"0094cc87-2dc6-4827-8288-a3406bf56018\") " Jul 07 16:24:13.810786 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:24:13.810618 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0094cc87-2dc6-4827-8288-a3406bf56018-workload-certs\") pod \"0094cc87-2dc6-4827-8288-a3406bf56018\" (UID: \"0094cc87-2dc6-4827-8288-a3406bf56018\") " Jul 07 16:24:13.810786 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:24:13.810669 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0094cc87-2dc6-4827-8288-a3406bf56018-credential-socket\") pod \"0094cc87-2dc6-4827-8288-a3406bf56018\" (UID: \"0094cc87-2dc6-4827-8288-a3406bf56018\") " Jul 07 16:24:13.810786 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:24:13.810713 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jpn88\" (UniqueName: \"kubernetes.io/projected/0094cc87-2dc6-4827-8288-a3406bf56018-kube-api-access-jpn88\") pod \"0094cc87-2dc6-4827-8288-a3406bf56018\" (UID: \"0094cc87-2dc6-4827-8288-a3406bf56018\") " Jul 07 16:24:13.810786 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:24:13.810771 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0094cc87-2dc6-4827-8288-a3406bf56018-istio-data\") pod \"0094cc87-2dc6-4827-8288-a3406bf56018\" (UID: \"0094cc87-2dc6-4827-8288-a3406bf56018\") " Jul 07 16:24:13.810993 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:24:13.810804 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0094cc87-2dc6-4827-8288-a3406bf56018-istio-podinfo\") pod \"0094cc87-2dc6-4827-8288-a3406bf56018\" (UID: \"0094cc87-2dc6-4827-8288-a3406bf56018\") " Jul 07 16:24:13.810993 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:24:13.810895 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0094cc87-2dc6-4827-8288-a3406bf56018-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "0094cc87-2dc6-4827-8288-a3406bf56018" (UID: "0094cc87-2dc6-4827-8288-a3406bf56018"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 16:24:13.810993 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:24:13.810936 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0094cc87-2dc6-4827-8288-a3406bf56018-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "0094cc87-2dc6-4827-8288-a3406bf56018" (UID: "0094cc87-2dc6-4827-8288-a3406bf56018"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 16:24:13.810993 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:24:13.810959 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0094cc87-2dc6-4827-8288-a3406bf56018-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "0094cc87-2dc6-4827-8288-a3406bf56018" (UID: "0094cc87-2dc6-4827-8288-a3406bf56018"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 16:24:13.811211 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:24:13.811183 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0094cc87-2dc6-4827-8288-a3406bf56018-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "0094cc87-2dc6-4827-8288-a3406bf56018" (UID: "0094cc87-2dc6-4827-8288-a3406bf56018"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 16:24:13.811276 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:24:13.811242 2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0094cc87-2dc6-4827-8288-a3406bf56018-istiod-ca-cert\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 07 16:24:13.811276 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:24:13.811257 2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0094cc87-2dc6-4827-8288-a3406bf56018-workload-socket\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 07 16:24:13.811276 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:24:13.811266 2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0094cc87-2dc6-4827-8288-a3406bf56018-workload-certs\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 07 16:24:13.811681 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:24:13.811661 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0094cc87-2dc6-4827-8288-a3406bf56018-istio-data" (OuterVolumeSpecName: "istio-data") pod "0094cc87-2dc6-4827-8288-a3406bf56018" (UID: "0094cc87-2dc6-4827-8288-a3406bf56018"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 16:24:13.812780 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:24:13.812757 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0094cc87-2dc6-4827-8288-a3406bf56018-istio-token" (OuterVolumeSpecName: "istio-token") pod "0094cc87-2dc6-4827-8288-a3406bf56018" (UID: "0094cc87-2dc6-4827-8288-a3406bf56018"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 16:24:13.812940 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:24:13.812923 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0094cc87-2dc6-4827-8288-a3406bf56018-kube-api-access-jpn88" (OuterVolumeSpecName: "kube-api-access-jpn88") pod "0094cc87-2dc6-4827-8288-a3406bf56018" (UID: "0094cc87-2dc6-4827-8288-a3406bf56018"). InnerVolumeSpecName "kube-api-access-jpn88". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 16:24:13.813086 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:24:13.813065 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0094cc87-2dc6-4827-8288-a3406bf56018-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "0094cc87-2dc6-4827-8288-a3406bf56018" (UID: "0094cc87-2dc6-4827-8288-a3406bf56018"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 16:24:13.813177 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:24:13.813101 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0094cc87-2dc6-4827-8288-a3406bf56018-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "0094cc87-2dc6-4827-8288-a3406bf56018" (UID: "0094cc87-2dc6-4827-8288-a3406bf56018"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 07 16:24:13.912591 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:24:13.912552 2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0094cc87-2dc6-4827-8288-a3406bf56018-credential-socket\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 07 16:24:13.912591 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:24:13.912583 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jpn88\" (UniqueName: \"kubernetes.io/projected/0094cc87-2dc6-4827-8288-a3406bf56018-kube-api-access-jpn88\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 07 16:24:13.912591 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:24:13.912593 2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0094cc87-2dc6-4827-8288-a3406bf56018-istio-data\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 07 16:24:13.912591 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:24:13.912602 2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0094cc87-2dc6-4827-8288-a3406bf56018-istio-podinfo\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 07 16:24:13.912856 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:24:13.912610 2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0094cc87-2dc6-4827-8288-a3406bf56018-istio-envoy\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 07 16:24:13.912856 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:24:13.912618 2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0094cc87-2dc6-4827-8288-a3406bf56018-istio-token\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 07 16:24:14.217731 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:24:14.217636 2565 generic.go:356] "Generic (PLEG): container finished" podID="0094cc87-2dc6-4827-8288-a3406bf56018" containerID="23b972ae92f97d0609794dae59eb2b6af58ebc96230ce1133bb4d798df632697" exitCode=0 Jul 07 16:24:14.217731 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:24:14.217685 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-382396-openshift-default-66d7949b4c-qq7rq" event={"ID":"0094cc87-2dc6-4827-8288-a3406bf56018","Type":"ContainerDied","Data":"23b972ae92f97d0609794dae59eb2b6af58ebc96230ce1133bb4d798df632697"} Jul 07 16:24:14.217731 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:24:14.217690 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-382396-openshift-default-66d7949b4c-qq7rq" Jul 07 16:24:14.217992 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:24:14.217737 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-382396-openshift-default-66d7949b4c-qq7rq" event={"ID":"0094cc87-2dc6-4827-8288-a3406bf56018","Type":"ContainerDied","Data":"97d1f7202068a79f46b14df25e67b112ae38f001116039b027a15b3a86f1a18f"} Jul 07 16:24:14.217992 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:24:14.217765 2565 scope.go:117] "RemoveContainer" containerID="23b972ae92f97d0609794dae59eb2b6af58ebc96230ce1133bb4d798df632697" Jul 07 16:24:14.227602 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:24:14.227581 2565 scope.go:117] "RemoveContainer" containerID="73f802b80f728d9a02c263972746bca5d3d091b4c8ab630f6f07c491d3abeafe" Jul 07 16:24:14.235762 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:24:14.235741 2565 scope.go:117] "RemoveContainer" containerID="23b972ae92f97d0609794dae59eb2b6af58ebc96230ce1133bb4d798df632697" Jul 07 16:24:14.236003 ip-10-0-132-65 kubenswrapper[2565]: E0707 16:24:14.235983 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"23b972ae92f97d0609794dae59eb2b6af58ebc96230ce1133bb4d798df632697\": container with ID starting with 23b972ae92f97d0609794dae59eb2b6af58ebc96230ce1133bb4d798df632697 not found: ID does not exist" containerID="23b972ae92f97d0609794dae59eb2b6af58ebc96230ce1133bb4d798df632697" Jul 07 16:24:14.236062 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:24:14.236010 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23b972ae92f97d0609794dae59eb2b6af58ebc96230ce1133bb4d798df632697"} err="failed to get container status \"23b972ae92f97d0609794dae59eb2b6af58ebc96230ce1133bb4d798df632697\": rpc error: code = NotFound desc = could not find container \"23b972ae92f97d0609794dae59eb2b6af58ebc96230ce1133bb4d798df632697\": container with ID starting with 23b972ae92f97d0609794dae59eb2b6af58ebc96230ce1133bb4d798df632697 not found: ID does not exist" Jul 07 16:24:14.236062 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:24:14.236027 2565 scope.go:117] "RemoveContainer" containerID="73f802b80f728d9a02c263972746bca5d3d091b4c8ab630f6f07c491d3abeafe" Jul 07 16:24:14.236243 ip-10-0-132-65 kubenswrapper[2565]: E0707 16:24:14.236229 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73f802b80f728d9a02c263972746bca5d3d091b4c8ab630f6f07c491d3abeafe\": container with ID starting with 73f802b80f728d9a02c263972746bca5d3d091b4c8ab630f6f07c491d3abeafe not found: ID does not exist" containerID="73f802b80f728d9a02c263972746bca5d3d091b4c8ab630f6f07c491d3abeafe" Jul 07 16:24:14.236283 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:24:14.236246 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73f802b80f728d9a02c263972746bca5d3d091b4c8ab630f6f07c491d3abeafe"} err="failed to get container status \"73f802b80f728d9a02c263972746bca5d3d091b4c8ab630f6f07c491d3abeafe\": rpc error: code = NotFound desc = could not find container \"73f802b80f728d9a02c263972746bca5d3d091b4c8ab630f6f07c491d3abeafe\": container with ID starting with 73f802b80f728d9a02c263972746bca5d3d091b4c8ab630f6f07c491d3abeafe not found: ID does not exist" Jul 07 16:24:14.242382 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:24:14.242360 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-382396-openshift-default-66d7949b4c-qq7rq"] Jul 07 16:24:14.246695 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:24:14.246676 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-382396-openshift-default-66d7949b4c-qq7rq"] Jul 07 16:24:15.660675 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:24:15.660632 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0094cc87-2dc6-4827-8288-a3406bf56018" path="/var/lib/kubelet/pods/0094cc87-2dc6-4827-8288-a3406bf56018/volumes" Jul 07 16:24:15.864838 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:24:15.864807 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:24:15.963083 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:24:15.963004 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:24:16.120737 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:24:16.120707 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-17e556-openshift-default-7dd7c6785f-f4bsw"] Jul 07 16:24:16.120909 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:24:16.120889 2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-17e556-openshift-default-7dd7c6785f-f4bsw" podUID="f039e558-63f8-49bf-a74e-2d05726d9444" containerName="istio-proxy" containerID="cri-o://6014ecbce46172a60ad73ab63575beda793cb6b91dc7e5b2865f01515c8ea827" gracePeriod=30 Jul 07 16:24:16.149164 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:24:16.149129 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:24:16.369720 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:24:16.369690 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:24:16.785568 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:24:16.783707 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-78c657d4d7-lrkgg_cb1758d6-32e9-4d5d-b9f8-7d3b1b16e303/authorino/0.log" Jul 07 16:24:21.255110 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:24:21.255085 2565 generic.go:356] "Generic (PLEG): container finished" podID="f039e558-63f8-49bf-a74e-2d05726d9444" containerID="6014ecbce46172a60ad73ab63575beda793cb6b91dc7e5b2865f01515c8ea827" exitCode=0 Jul 07 16:24:21.255486 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:24:21.255163 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-17e556-openshift-default-7dd7c6785f-f4bsw" event={"ID":"f039e558-63f8-49bf-a74e-2d05726d9444","Type":"ContainerDied","Data":"6014ecbce46172a60ad73ab63575beda793cb6b91dc7e5b2865f01515c8ea827"} Jul 07 16:24:21.255486 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:24:21.255240 2565 scope.go:117] "RemoveContainer" containerID="7f44bcc339fd0068c97dc6bbc388cd631f0cc6ee61279de95b49f5786baff1ad" Jul 07 16:24:21.385710 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:24:21.385649 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-17e556-openshift-default-7dd7c6785f-f4bsw" Jul 07 16:24:21.482247 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:24:21.482211 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f039e558-63f8-49bf-a74e-2d05726d9444-istio-envoy\") pod \"f039e558-63f8-49bf-a74e-2d05726d9444\" (UID: \"f039e558-63f8-49bf-a74e-2d05726d9444\") " Jul 07 16:24:21.482247 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:24:21.482250 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f039e558-63f8-49bf-a74e-2d05726d9444-istio-podinfo\") pod \"f039e558-63f8-49bf-a74e-2d05726d9444\" (UID: \"f039e558-63f8-49bf-a74e-2d05726d9444\") " Jul 07 16:24:21.482473 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:24:21.482283 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f039e558-63f8-49bf-a74e-2d05726d9444-workload-socket\") pod \"f039e558-63f8-49bf-a74e-2d05726d9444\" (UID: \"f039e558-63f8-49bf-a74e-2d05726d9444\") " Jul 07 16:24:21.482473 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:24:21.482332 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f039e558-63f8-49bf-a74e-2d05726d9444-credential-socket\") pod \"f039e558-63f8-49bf-a74e-2d05726d9444\" (UID: \"f039e558-63f8-49bf-a74e-2d05726d9444\") " Jul 07 16:24:21.482473 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:24:21.482363 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-52ndv\" (UniqueName: \"kubernetes.io/projected/f039e558-63f8-49bf-a74e-2d05726d9444-kube-api-access-52ndv\") pod \"f039e558-63f8-49bf-a74e-2d05726d9444\" (UID: \"f039e558-63f8-49bf-a74e-2d05726d9444\") " Jul 07 16:24:21.482473 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:24:21.482381 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f039e558-63f8-49bf-a74e-2d05726d9444-workload-certs\") pod \"f039e558-63f8-49bf-a74e-2d05726d9444\" (UID: \"f039e558-63f8-49bf-a74e-2d05726d9444\") " Jul 07 16:24:21.482473 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:24:21.482433 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f039e558-63f8-49bf-a74e-2d05726d9444-istio-token\") pod \"f039e558-63f8-49bf-a74e-2d05726d9444\" (UID: \"f039e558-63f8-49bf-a74e-2d05726d9444\") " Jul 07 16:24:21.482771 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:24:21.482479 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f039e558-63f8-49bf-a74e-2d05726d9444-istio-data\") pod \"f039e558-63f8-49bf-a74e-2d05726d9444\" (UID: \"f039e558-63f8-49bf-a74e-2d05726d9444\") " Jul 07 16:24:21.482771 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:24:21.482535 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f039e558-63f8-49bf-a74e-2d05726d9444-istiod-ca-cert\") pod \"f039e558-63f8-49bf-a74e-2d05726d9444\" (UID: \"f039e558-63f8-49bf-a74e-2d05726d9444\") " Jul 07 16:24:21.482771 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:24:21.482708 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f039e558-63f8-49bf-a74e-2d05726d9444-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "f039e558-63f8-49bf-a74e-2d05726d9444" (UID: "f039e558-63f8-49bf-a74e-2d05726d9444"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 16:24:21.482771 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:24:21.482730 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f039e558-63f8-49bf-a74e-2d05726d9444-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "f039e558-63f8-49bf-a74e-2d05726d9444" (UID: "f039e558-63f8-49bf-a74e-2d05726d9444"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 16:24:21.482980 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:24:21.482845 2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f039e558-63f8-49bf-a74e-2d05726d9444-credential-socket\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 07 16:24:21.482980 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:24:21.482864 2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f039e558-63f8-49bf-a74e-2d05726d9444-workload-certs\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 07 16:24:21.482980 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:24:21.482969 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f039e558-63f8-49bf-a74e-2d05726d9444-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "f039e558-63f8-49bf-a74e-2d05726d9444" (UID: "f039e558-63f8-49bf-a74e-2d05726d9444"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 16:24:21.483125 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:24:21.483064 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f039e558-63f8-49bf-a74e-2d05726d9444-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "f039e558-63f8-49bf-a74e-2d05726d9444" (UID: "f039e558-63f8-49bf-a74e-2d05726d9444"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 16:24:21.483450 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:24:21.483429 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f039e558-63f8-49bf-a74e-2d05726d9444-istio-data" (OuterVolumeSpecName: "istio-data") pod "f039e558-63f8-49bf-a74e-2d05726d9444" (UID: "f039e558-63f8-49bf-a74e-2d05726d9444"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 16:24:21.484538 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:24:21.484503 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f039e558-63f8-49bf-a74e-2d05726d9444-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "f039e558-63f8-49bf-a74e-2d05726d9444" (UID: "f039e558-63f8-49bf-a74e-2d05726d9444"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 07 16:24:21.484651 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:24:21.484587 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f039e558-63f8-49bf-a74e-2d05726d9444-kube-api-access-52ndv" (OuterVolumeSpecName: "kube-api-access-52ndv") pod "f039e558-63f8-49bf-a74e-2d05726d9444" (UID: "f039e558-63f8-49bf-a74e-2d05726d9444"). InnerVolumeSpecName "kube-api-access-52ndv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 16:24:21.484809 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:24:21.484790 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f039e558-63f8-49bf-a74e-2d05726d9444-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "f039e558-63f8-49bf-a74e-2d05726d9444" (UID: "f039e558-63f8-49bf-a74e-2d05726d9444"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 16:24:21.484914 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:24:21.484895 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f039e558-63f8-49bf-a74e-2d05726d9444-istio-token" (OuterVolumeSpecName: "istio-token") pod "f039e558-63f8-49bf-a74e-2d05726d9444" (UID: "f039e558-63f8-49bf-a74e-2d05726d9444"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 16:24:21.570433 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:24:21.570404 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-gz46p_8a5048fd-5651-491f-9cea-7f4d93706202/manager/0.log" Jul 07 16:24:21.584172 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:24:21.584143 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-52ndv\" (UniqueName: \"kubernetes.io/projected/f039e558-63f8-49bf-a74e-2d05726d9444-kube-api-access-52ndv\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 07 16:24:21.584172 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:24:21.584168 2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f039e558-63f8-49bf-a74e-2d05726d9444-istio-token\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 07 16:24:21.584352 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:24:21.584180 2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f039e558-63f8-49bf-a74e-2d05726d9444-istio-data\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 07 16:24:21.584352 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:24:21.584189 2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f039e558-63f8-49bf-a74e-2d05726d9444-istiod-ca-cert\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 07 16:24:21.584352 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:24:21.584196 2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f039e558-63f8-49bf-a74e-2d05726d9444-istio-envoy\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 07 16:24:21.584352 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:24:21.584204 2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f039e558-63f8-49bf-a74e-2d05726d9444-istio-podinfo\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 07 16:24:21.584352 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:24:21.584212 2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f039e558-63f8-49bf-a74e-2d05726d9444-workload-socket\") on node \"ip-10-0-132-65.ec2.internal\" DevicePath \"\"" Jul 07 16:24:21.694918 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:24:21.694887 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-l6mcc_adac090f-b466-4ea0-b254-ec2d65dd63af/manager/0.log" Jul 07 16:24:21.812266 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:24:21.812234 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-api-5f874bfc85-qc79p_5ae76816-ef6b-4546-9b65-571e4a89d035/maas-api/0.log" Jul 07 16:24:21.927087 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:24:21.927007 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-api-key-cleanup-29724000-2m42d_1669c122-070a-4df8-876d-dac9e83c14f2/cleanup/0.log" Jul 07 16:24:22.195434 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:24:22.195357 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-586d4c4bbb-knwts_a399b6c7-1bdf-47da-a5ee-6a3251cfc136/manager/0.log" Jul 07 16:24:22.260806 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:24:22.260782 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-17e556-openshift-default-7dd7c6785f-f4bsw" Jul 07 16:24:22.261275 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:24:22.260779 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-17e556-openshift-default-7dd7c6785f-f4bsw" event={"ID":"f039e558-63f8-49bf-a74e-2d05726d9444","Type":"ContainerDied","Data":"841f94208770d0cf602657eedd80041cbf6e207a543b5064b3c1118ab7b10444"} Jul 07 16:24:22.261275 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:24:22.260905 2565 scope.go:117] "RemoveContainer" containerID="6014ecbce46172a60ad73ab63575beda793cb6b91dc7e5b2865f01515c8ea827" Jul 07 16:24:22.286360 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:24:22.286329 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-17e556-openshift-default-7dd7c6785f-f4bsw"] Jul 07 16:24:22.290805 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:24:22.290781 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-17e556-openshift-default-7dd7c6785f-f4bsw"] Jul 07 16:24:22.311783 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:24:22.311763 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-q85t9_338a6794-f166-47b8-929e-14332af6e1bf/server/0.log" Jul 07 16:24:22.438921 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:24:22.438896 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-4wkpn_96ac3c97-7a14-441e-a092-3cd28b8c2b13/manager/0.log" Jul 07 16:24:22.820041 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:24:22.820014 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-685dff6fcf-lmkgr_fb691f38-96cb-4df1-bdaf-682a080bff74/copy-manifests/0.log" Jul 07 16:24:22.833591 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:24:22.833556 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-685dff6fcf-lmkgr_fb691f38-96cb-4df1-bdaf-682a080bff74/manager/0.log" Jul 07 16:24:22.949152 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:24:22.949127 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_postgres-69fb4594cf-wrv28_3fdac274-a101-4e09-9e15-83842c752d99/postgres/0.log" Jul 07 16:24:23.660235 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:24:23.660154 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f039e558-63f8-49bf-a74e-2d05726d9444" path="/var/lib/kubelet/pods/f039e558-63f8-49bf-a74e-2d05726d9444/volumes" Jul 07 16:24:24.285348 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:24:24.285317 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-78c657d4d7-lrkgg_cb1758d6-32e9-4d5d-b9f8-7d3b1b16e303/authorino/0.log" Jul 07 16:24:24.406301 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:24:24.406273 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-wvkvz_6957a016-f9f9-4ec4-bc6a-1e8edc3ed6fa/manager/0.log" Jul 07 16:24:24.524719 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:24:24.524693 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-m8d5b_2c9e0255-074e-4dcc-a8e8-ae664d60d2bb/manager/0.log" Jul 07 16:24:24.755205 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:24:24.755170 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-6fszm_62427603-a6ea-4c7d-98ee-347671a9d227/registry-server/0.log" Jul 07 16:24:25.150593 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:24:25.150562 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-nx6zw_6555056e-fa42-4326-becf-9bd8eb0cfc21/manager/0.log" Jul 07 16:24:25.507695 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:24:25.507620 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7f964fb678k74l4_a5c36a68-d3a9-492c-a364-91b5785b16c1/istio-proxy/0.log" Jul 07 16:24:25.863855 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:24:25.863826 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-ccc59c597-jlq44_21bfa646-dd2e-40c2-847c-139c13e56c53/kube-auth-proxy/0.log" Jul 07 16:24:25.986705 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:24:25.986677 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-tqwxj_ea743bfe-7a54-46a1-8f15-cb6e93ee0f98/istio-proxy/1.log" Jul 07 16:24:26.339846 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:24:26.339813 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6b6cdf54ff-tjwbr_95b55bd6-e205-4969-ac44-d3f67f63f746/router/0.log" Jul 07 16:24:26.704384 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:24:26.704313 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-pcsd8_ed802748-bb8a-410e-bc9b-c342536171c3/main/0.log" Jul 07 16:24:27.069573 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:24:27.069541 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbsqs7b_2bc80ed7-bc86-4e0d-8347-4b61a0804f42/main/0.log" Jul 07 16:24:31.877334 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:24:31.877304 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-kfm9g/must-gather-fnvmg"] Jul 07 16:24:31.877740 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:24:31.877689 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f039e558-63f8-49bf-a74e-2d05726d9444" containerName="istio-proxy" Jul 07 16:24:31.877740 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:24:31.877701 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="f039e558-63f8-49bf-a74e-2d05726d9444" containerName="istio-proxy" Jul 07 16:24:31.877740 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:24:31.877714 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="96a93e1b-b1cb-4f98-bf0e-f134045ea199" containerName="main" Jul 07 16:24:31.877740 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:24:31.877719 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="96a93e1b-b1cb-4f98-bf0e-f134045ea199" containerName="main" Jul 07 16:24:31.877740 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:24:31.877736 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0094cc87-2dc6-4827-8288-a3406bf56018" containerName="istio-proxy" Jul 07 16:24:31.877740 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:24:31.877742 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="0094cc87-2dc6-4827-8288-a3406bf56018" containerName="istio-proxy" Jul 07 16:24:31.877941 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:24:31.877811 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="96a93e1b-b1cb-4f98-bf0e-f134045ea199" containerName="main" Jul 07 16:24:31.877941 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:24:31.877819 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="0094cc87-2dc6-4827-8288-a3406bf56018" containerName="istio-proxy" Jul 07 16:24:31.877941 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:24:31.877826 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="f039e558-63f8-49bf-a74e-2d05726d9444" containerName="istio-proxy" Jul 07 16:24:31.877941 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:24:31.877834 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="0094cc87-2dc6-4827-8288-a3406bf56018" containerName="istio-proxy" Jul 07 16:24:31.877941 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:24:31.877839 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="f039e558-63f8-49bf-a74e-2d05726d9444" containerName="istio-proxy" Jul 07 16:24:31.877941 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:24:31.877893 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0094cc87-2dc6-4827-8288-a3406bf56018" containerName="istio-proxy" Jul 07 16:24:31.877941 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:24:31.877899 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="0094cc87-2dc6-4827-8288-a3406bf56018" containerName="istio-proxy" Jul 07 16:24:31.877941 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:24:31.877914 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f039e558-63f8-49bf-a74e-2d05726d9444" containerName="istio-proxy" Jul 07 16:24:31.877941 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:24:31.877919 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="f039e558-63f8-49bf-a74e-2d05726d9444" containerName="istio-proxy" Jul 07 16:24:31.882359 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:24:31.882343 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-kfm9g/must-gather-fnvmg" Jul 07 16:24:31.885592 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:24:31.885559 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-kfm9g\"/\"kube-root-ca.crt\"" Jul 07 16:24:31.886590 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:24:31.886563 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:24:31.886693 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:24:31.886672 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-kfm9g\"/\"default-dockercfg-pc4lw\"" Jul 07 16:24:31.886744 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:24:31.886672 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-kfm9g\"/\"openshift-service-ca.crt\"" Jul 07 16:24:31.891788 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:24:31.891769 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-kfm9g/must-gather-fnvmg"] Jul 07 16:24:31.972447 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:24:31.972402 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbnmw\" (UniqueName: \"kubernetes.io/projected/7e4dc0e5-836f-499c-8afa-91ced03b026f-kube-api-access-sbnmw\") pod \"must-gather-fnvmg\" (UID: \"7e4dc0e5-836f-499c-8afa-91ced03b026f\") " pod="openshift-must-gather-kfm9g/must-gather-fnvmg" Jul 07 16:24:31.972447 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:24:31.972446 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/7e4dc0e5-836f-499c-8afa-91ced03b026f-must-gather-output\") pod \"must-gather-fnvmg\" (UID: \"7e4dc0e5-836f-499c-8afa-91ced03b026f\") " pod="openshift-must-gather-kfm9g/must-gather-fnvmg" Jul 07 16:24:31.982462 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:24:31.982436 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" limit=1024 Jul 07 16:24:32.073248 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:24:32.073217 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbnmw\" (UniqueName: \"kubernetes.io/projected/7e4dc0e5-836f-499c-8afa-91ced03b026f-kube-api-access-sbnmw\") pod \"must-gather-fnvmg\" (UID: \"7e4dc0e5-836f-499c-8afa-91ced03b026f\") " pod="openshift-must-gather-kfm9g/must-gather-fnvmg" Jul 07 16:24:32.073248 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:24:32.073252 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/7e4dc0e5-836f-499c-8afa-91ced03b026f-must-gather-output\") pod \"must-gather-fnvmg\" (UID: \"7e4dc0e5-836f-499c-8afa-91ced03b026f\") " pod="openshift-must-gather-kfm9g/must-gather-fnvmg" Jul 07 16:24:32.073618 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:24:32.073601 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/7e4dc0e5-836f-499c-8afa-91ced03b026f-must-gather-output\") pod \"must-gather-fnvmg\" (UID: \"7e4dc0e5-836f-499c-8afa-91ced03b026f\") " pod="openshift-must-gather-kfm9g/must-gather-fnvmg" Jul 07 16:24:32.083192 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:24:32.083166 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbnmw\" (UniqueName: \"kubernetes.io/projected/7e4dc0e5-836f-499c-8afa-91ced03b026f-kube-api-access-sbnmw\") pod \"must-gather-fnvmg\" (UID: \"7e4dc0e5-836f-499c-8afa-91ced03b026f\") " pod="openshift-must-gather-kfm9g/must-gather-fnvmg" Jul 07 16:24:32.192681 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:24:32.192592 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-kfm9g/must-gather-fnvmg" Jul 07 16:24:32.331837 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:24:32.331805 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-kfm9g/must-gather-fnvmg"] Jul 07 16:24:32.333346 ip-10-0-132-65 kubenswrapper[2565]: W0707 16:24:32.333322 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7e4dc0e5_836f_499c_8afa_91ced03b026f.slice/crio-9fee3530bc1e42353825b3d5cadea3e350706553f0748873829c26f11f0ff90e WatchSource:0}: Error finding container 9fee3530bc1e42353825b3d5cadea3e350706553f0748873829c26f11f0ff90e: Status 404 returned error can't find the container with id 9fee3530bc1e42353825b3d5cadea3e350706553f0748873829c26f11f0ff90e Jul 07 16:24:32.335200 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:24:32.335185 2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 07 16:24:33.316830 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:24:33.316797 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kfm9g/must-gather-fnvmg" event={"ID":"7e4dc0e5-836f-499c-8afa-91ced03b026f","Type":"ContainerStarted","Data":"569a98ca34f864247e93e53d22a1c0f0370b54d3d6a53be2956cadfd3e0cd453"} Jul 07 16:24:33.317261 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:24:33.316838 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kfm9g/must-gather-fnvmg" event={"ID":"7e4dc0e5-836f-499c-8afa-91ced03b026f","Type":"ContainerStarted","Data":"5af945141a9ac19a6634de7a544fbc44d66681cccdf6c900a656c3d555019585"} Jul 07 16:24:33.317261 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:24:33.316850 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kfm9g/must-gather-fnvmg" event={"ID":"7e4dc0e5-836f-499c-8afa-91ced03b026f","Type":"ContainerStarted","Data":"9fee3530bc1e42353825b3d5cadea3e350706553f0748873829c26f11f0ff90e"} Jul 07 16:24:33.339024 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:24:33.338975 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-kfm9g/must-gather-fnvmg" podStartSLOduration=1.641424794 podStartE2EDuration="2.338961111s" totalImagesPullingTime="697.536317ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 16:24:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:24:33.337204227 +0000 UTC m=+2612.293781695" watchObservedRunningTime="2026-07-07 16:24:33.338961111 +0000 UTC m=+2612.295538577" Jul 07 16:24:34.759755 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:24:34.759727 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-6qzkn_0c5071d6-cead-4992-a7e9-9a752b955cae/global-pull-secret-syncer/0.log" Jul 07 16:24:34.916936 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:24:34.916899 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-m4wtp_e6634bed-1f16-4667-ae9e-c08984716ce5/konnectivity-agent/0.log" Jul 07 16:24:35.035847 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:24:35.035766 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-132-65.ec2.internal_9fc4f78a01c7da970c3b1d6baab931c6/haproxy/0.log" Jul 07 16:24:40.047517 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:24:40.047357 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-78c657d4d7-lrkgg_cb1758d6-32e9-4d5d-b9f8-7d3b1b16e303/authorino/0.log" Jul 07 16:24:40.079362 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:24:40.079322 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-wvkvz_6957a016-f9f9-4ec4-bc6a-1e8edc3ed6fa/manager/0.log" Jul 07 16:24:40.109031 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:24:40.109004 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-m8d5b_2c9e0255-074e-4dcc-a8e8-ae664d60d2bb/manager/0.log" Jul 07 16:24:40.220052 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:24:40.220015 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-6fszm_62427603-a6ea-4c7d-98ee-347671a9d227/registry-server/0.log" Jul 07 16:24:40.502097 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:24:40.502064 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-nx6zw_6555056e-fa42-4326-becf-9bd8eb0cfc21/manager/0.log" Jul 07 16:24:42.613734 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:24:42.613706 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-69d5v_767860ff-f41a-4b9b-962c-f0083d0f069a/node-exporter/0.log" Jul 07 16:24:42.642383 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:24:42.642357 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-69d5v_767860ff-f41a-4b9b-962c-f0083d0f069a/kube-rbac-proxy/0.log" Jul 07 16:24:42.670416 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:24:42.670389 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-69d5v_767860ff-f41a-4b9b-962c-f0083d0f069a/init-textfile/0.log" Jul 07 16:24:44.682184 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:24:44.682142 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-kfm9g/perf-node-gather-daemonset-qsz9n"] Jul 07 16:24:44.689867 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:24:44.689837 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-kfm9g/perf-node-gather-daemonset-qsz9n" Jul 07 16:24:44.693953 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:24:44.693921 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-kfm9g/perf-node-gather-daemonset-qsz9n"] Jul 07 16:24:44.808694 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:24:44.808660 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1457f9c9-b05a-4ff5-8743-d044937b83b7-lib-modules\") pod \"perf-node-gather-daemonset-qsz9n\" (UID: \"1457f9c9-b05a-4ff5-8743-d044937b83b7\") " pod="openshift-must-gather-kfm9g/perf-node-gather-daemonset-qsz9n" Jul 07 16:24:44.808883 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:24:44.808735 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bz9pp\" (UniqueName: \"kubernetes.io/projected/1457f9c9-b05a-4ff5-8743-d044937b83b7-kube-api-access-bz9pp\") pod \"perf-node-gather-daemonset-qsz9n\" (UID: \"1457f9c9-b05a-4ff5-8743-d044937b83b7\") " pod="openshift-must-gather-kfm9g/perf-node-gather-daemonset-qsz9n" Jul 07 16:24:44.808883 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:24:44.808782 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1457f9c9-b05a-4ff5-8743-d044937b83b7-sys\") pod \"perf-node-gather-daemonset-qsz9n\" (UID: \"1457f9c9-b05a-4ff5-8743-d044937b83b7\") " pod="openshift-must-gather-kfm9g/perf-node-gather-daemonset-qsz9n" Jul 07 16:24:44.808883 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:24:44.808817 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/1457f9c9-b05a-4ff5-8743-d044937b83b7-podres\") pod \"perf-node-gather-daemonset-qsz9n\" (UID: \"1457f9c9-b05a-4ff5-8743-d044937b83b7\") " pod="openshift-must-gather-kfm9g/perf-node-gather-daemonset-qsz9n" Jul 07 16:24:44.809040 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:24:44.808961 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/1457f9c9-b05a-4ff5-8743-d044937b83b7-proc\") pod \"perf-node-gather-daemonset-qsz9n\" (UID: \"1457f9c9-b05a-4ff5-8743-d044937b83b7\") " pod="openshift-must-gather-kfm9g/perf-node-gather-daemonset-qsz9n" Jul 07 16:24:44.818520 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:24:44.818478 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-6bffdf9778-wz5k7_0cbd6022-2c42-43b2-950f-ab089797a659/networking-console-plugin/0.log" Jul 07 16:24:44.910558 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:24:44.910518 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/1457f9c9-b05a-4ff5-8743-d044937b83b7-podres\") pod \"perf-node-gather-daemonset-qsz9n\" (UID: \"1457f9c9-b05a-4ff5-8743-d044937b83b7\") " pod="openshift-must-gather-kfm9g/perf-node-gather-daemonset-qsz9n" Jul 07 16:24:44.910887 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:24:44.910866 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/1457f9c9-b05a-4ff5-8743-d044937b83b7-proc\") pod \"perf-node-gather-daemonset-qsz9n\" (UID: \"1457f9c9-b05a-4ff5-8743-d044937b83b7\") " pod="openshift-must-gather-kfm9g/perf-node-gather-daemonset-qsz9n" Jul 07 16:24:44.911024 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:24:44.911007 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1457f9c9-b05a-4ff5-8743-d044937b83b7-lib-modules\") pod \"perf-node-gather-daemonset-qsz9n\" (UID: \"1457f9c9-b05a-4ff5-8743-d044937b83b7\") " pod="openshift-must-gather-kfm9g/perf-node-gather-daemonset-qsz9n" Jul 07 16:24:44.911172 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:24:44.911156 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bz9pp\" (UniqueName: \"kubernetes.io/projected/1457f9c9-b05a-4ff5-8743-d044937b83b7-kube-api-access-bz9pp\") pod \"perf-node-gather-daemonset-qsz9n\" (UID: \"1457f9c9-b05a-4ff5-8743-d044937b83b7\") " pod="openshift-must-gather-kfm9g/perf-node-gather-daemonset-qsz9n" Jul 07 16:24:44.911307 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:24:44.911292 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1457f9c9-b05a-4ff5-8743-d044937b83b7-sys\") pod \"perf-node-gather-daemonset-qsz9n\" (UID: \"1457f9c9-b05a-4ff5-8743-d044937b83b7\") " pod="openshift-must-gather-kfm9g/perf-node-gather-daemonset-qsz9n" Jul 07 16:24:44.911466 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:24:44.910962 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/1457f9c9-b05a-4ff5-8743-d044937b83b7-proc\") pod \"perf-node-gather-daemonset-qsz9n\" (UID: \"1457f9c9-b05a-4ff5-8743-d044937b83b7\") " pod="openshift-must-gather-kfm9g/perf-node-gather-daemonset-qsz9n" Jul 07 16:24:44.911567 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:24:44.910901 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/1457f9c9-b05a-4ff5-8743-d044937b83b7-podres\") pod \"perf-node-gather-daemonset-qsz9n\" (UID: \"1457f9c9-b05a-4ff5-8743-d044937b83b7\") " pod="openshift-must-gather-kfm9g/perf-node-gather-daemonset-qsz9n" Jul 07 16:24:44.911636 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:24:44.911581 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1457f9c9-b05a-4ff5-8743-d044937b83b7-lib-modules\") pod \"perf-node-gather-daemonset-qsz9n\" (UID: \"1457f9c9-b05a-4ff5-8743-d044937b83b7\") " pod="openshift-must-gather-kfm9g/perf-node-gather-daemonset-qsz9n" Jul 07 16:24:44.911685 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:24:44.911636 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1457f9c9-b05a-4ff5-8743-d044937b83b7-sys\") pod \"perf-node-gather-daemonset-qsz9n\" (UID: \"1457f9c9-b05a-4ff5-8743-d044937b83b7\") " pod="openshift-must-gather-kfm9g/perf-node-gather-daemonset-qsz9n" Jul 07 16:24:44.923252 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:24:44.923221 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bz9pp\" (UniqueName: \"kubernetes.io/projected/1457f9c9-b05a-4ff5-8743-d044937b83b7-kube-api-access-bz9pp\") pod \"perf-node-gather-daemonset-qsz9n\" (UID: \"1457f9c9-b05a-4ff5-8743-d044937b83b7\") " pod="openshift-must-gather-kfm9g/perf-node-gather-daemonset-qsz9n" Jul 07 16:24:45.005828 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:24:45.005744 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-kfm9g/perf-node-gather-daemonset-qsz9n" Jul 07 16:24:45.184717 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:24:45.184556 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-kfm9g/perf-node-gather-daemonset-qsz9n"] Jul 07 16:24:45.188402 ip-10-0-132-65 kubenswrapper[2565]: W0707 16:24:45.188364 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod1457f9c9_b05a_4ff5_8743_d044937b83b7.slice/crio-731d85c12820218cc462292d6b18f26caa658856ec1c76d26aa2e04255c62749 WatchSource:0}: Error finding container 731d85c12820218cc462292d6b18f26caa658856ec1c76d26aa2e04255c62749: Status 404 returned error can't find the container with id 731d85c12820218cc462292d6b18f26caa658856ec1c76d26aa2e04255c62749 Jul 07 16:24:45.409619 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:24:45.409575 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kfm9g/perf-node-gather-daemonset-qsz9n" event={"ID":"1457f9c9-b05a-4ff5-8743-d044937b83b7","Type":"ContainerStarted","Data":"f544da0f68761da924c048dbceaa869ab1231781a8eef3402966035006acb5e4"} Jul 07 16:24:45.409809 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:24:45.409631 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kfm9g/perf-node-gather-daemonset-qsz9n" event={"ID":"1457f9c9-b05a-4ff5-8743-d044937b83b7","Type":"ContainerStarted","Data":"731d85c12820218cc462292d6b18f26caa658856ec1c76d26aa2e04255c62749"} Jul 07 16:24:45.409809 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:24:45.409738 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-kfm9g/perf-node-gather-daemonset-qsz9n" Jul 07 16:24:45.432890 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:24:45.432832 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-kfm9g/perf-node-gather-daemonset-qsz9n" podStartSLOduration=1.432810512 podStartE2EDuration="1.432810512s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 16:24:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 16:24:45.430530408 +0000 UTC m=+2624.387107872" watchObservedRunningTime="2026-07-07 16:24:45.432810512 +0000 UTC m=+2624.389387980" Jul 07 16:24:45.451621 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:24:45.451596 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-s9kl8_62da0165-d32c-4bba-9844-a49154d765e5/console-operator/1.log" Jul 07 16:24:45.456234 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:24:45.456207 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-s9kl8_62da0165-d32c-4bba-9844-a49154d765e5/console-operator/2.log" Jul 07 16:24:47.944261 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:24:47.944234 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-qhqsf_cb4ddf17-3bbe-4d75-9510-99f4a61da167/dns/0.log" Jul 07 16:24:47.974375 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:24:47.974345 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-qhqsf_cb4ddf17-3bbe-4d75-9510-99f4a61da167/kube-rbac-proxy/0.log" Jul 07 16:24:48.071335 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:24:48.071309 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-qggz6_d6c50a3b-0739-431c-80b0-8f42c4b00405/dns-node-resolver/0.log" Jul 07 16:24:48.667572 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:24:48.667542 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-7f7d849b7d-bvxw4_f9bae27f-4189-4ec5-9f1f-b2036270ebe3/registry/0.log" Jul 07 16:24:48.693028 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:24:48.692992 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-9fmf8_e5c3e573-02fc-4e63-b475-65ee8b00f34b/node-ca/0.log" Jul 07 16:24:49.790407 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:24:49.790383 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7f964fb678k74l4_a5c36a68-d3a9-492c-a364-91b5785b16c1/istio-proxy/0.log" Jul 07 16:24:49.968079 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:24:49.968044 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-ccc59c597-jlq44_21bfa646-dd2e-40c2-847c-139c13e56c53/kube-auth-proxy/0.log" Jul 07 16:24:50.024136 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:24:50.024108 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-tqwxj_ea743bfe-7a54-46a1-8f15-cb6e93ee0f98/istio-proxy/0.log" Jul 07 16:24:50.058927 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:24:50.058848 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-tqwxj_ea743bfe-7a54-46a1-8f15-cb6e93ee0f98/istio-proxy/1.log" Jul 07 16:24:50.172577 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:24:50.172546 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6b6cdf54ff-tjwbr_95b55bd6-e205-4969-ac44-d3f67f63f746/router/0.log" Jul 07 16:24:50.970156 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:24:50.970127 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-t6xpc_4097f76e-a65f-4fa3-a14f-491cd0d5ce3f/serve-healthcheck-canary/0.log" Jul 07 16:24:51.429534 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:24:51.429451 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-kfm9g/perf-node-gather-daemonset-qsz9n" Jul 07 16:24:51.485324 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:24:51.485287 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5c847dcfc4-m4jd9_17238627-b0d8-49ae-827e-cbf250ab8146/insights-operator/0.log" Jul 07 16:24:51.486024 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:24:51.486004 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5c847dcfc4-m4jd9_17238627-b0d8-49ae-827e-cbf250ab8146/insights-operator/1.log" Jul 07 16:24:51.684479 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:24:51.684393 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-vthmh_12101a9b-a592-4514-9eaf-513f0feeb2e9/kube-rbac-proxy/0.log" Jul 07 16:24:51.714098 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:24:51.714063 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-vthmh_12101a9b-a592-4514-9eaf-513f0feeb2e9/exporter/0.log" Jul 07 16:24:51.742335 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:24:51.742310 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-vthmh_12101a9b-a592-4514-9eaf-513f0feeb2e9/extractor/0.log" Jul 07 16:24:54.294562 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:24:54.294453 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-gz46p_8a5048fd-5651-491f-9cea-7f4d93706202/manager/0.log" Jul 07 16:24:54.500271 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:24:54.500229 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-l6mcc_adac090f-b466-4ea0-b254-ec2d65dd63af/manager/0.log" Jul 07 16:24:54.551268 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:24:54.551193 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-api-5f874bfc85-qc79p_5ae76816-ef6b-4546-9b65-571e4a89d035/maas-api/0.log" Jul 07 16:24:54.580263 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:24:54.580238 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-api-key-cleanup-29724000-2m42d_1669c122-070a-4df8-876d-dac9e83c14f2/cleanup/0.log" Jul 07 16:24:54.761289 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:24:54.761223 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-586d4c4bbb-knwts_a399b6c7-1bdf-47da-a5ee-6a3251cfc136/manager/0.log" Jul 07 16:24:54.791440 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:24:54.791414 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-q85t9_338a6794-f166-47b8-929e-14332af6e1bf/server/0.log" Jul 07 16:24:54.902480 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:24:54.902451 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-4wkpn_96ac3c97-7a14-441e-a092-3cd28b8c2b13/manager/0.log" Jul 07 16:24:55.118601 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:24:55.118572 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-685dff6fcf-lmkgr_fb691f38-96cb-4df1-bdaf-682a080bff74/manager/0.log" Jul 07 16:24:55.161444 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:24:55.161356 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-685dff6fcf-lmkgr_fb691f38-96cb-4df1-bdaf-682a080bff74/copy-manifests/0.log" Jul 07 16:24:55.209370 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:24:55.209332 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_postgres-69fb4594cf-wrv28_3fdac274-a101-4e09-9e15-83842c752d99/postgres/0.log" Jul 07 16:24:55.286814 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:24:55.286780 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7f964fb678k74l4_a5c36a68-d3a9-492c-a364-91b5785b16c1/istio-proxy/0.log" Jul 07 16:24:55.448978 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:24:55.448891 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-ccc59c597-jlq44_21bfa646-dd2e-40c2-847c-139c13e56c53/kube-auth-proxy/0.log" Jul 07 16:24:55.471478 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:24:55.471449 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-tqwxj_ea743bfe-7a54-46a1-8f15-cb6e93ee0f98/istio-proxy/0.log" Jul 07 16:24:55.509550 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:24:55.509519 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-tqwxj_ea743bfe-7a54-46a1-8f15-cb6e93ee0f98/istio-proxy/1.log" Jul 07 16:24:55.584707 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:24:55.584668 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6b6cdf54ff-tjwbr_95b55bd6-e205-4969-ac44-d3f67f63f746/router/0.log" Jul 07 16:24:56.819586 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:24:56.819556 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7f964fb678k74l4_a5c36a68-d3a9-492c-a364-91b5785b16c1/istio-proxy/0.log" Jul 07 16:24:56.961326 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:24:56.961294 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-ccc59c597-jlq44_21bfa646-dd2e-40c2-847c-139c13e56c53/kube-auth-proxy/0.log" Jul 07 16:24:56.991068 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:24:56.990983 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-tqwxj_ea743bfe-7a54-46a1-8f15-cb6e93ee0f98/istio-proxy/0.log" Jul 07 16:24:57.022944 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:24:57.022910 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-tqwxj_ea743bfe-7a54-46a1-8f15-cb6e93ee0f98/istio-proxy/1.log" Jul 07 16:24:57.101247 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:24:57.101214 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6b6cdf54ff-tjwbr_95b55bd6-e205-4969-ac44-d3f67f63f746/router/0.log" Jul 07 16:25:01.765714 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:25:01.765636 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-d57d58cc9-l8sjd_0fe973cf-559f-4351-86fc-308c879b0541/migrator/0.log" Jul 07 16:25:01.794076 ip-10-0-132-65 kubenswrapper[2565]: I0707 16:25:01.794038 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-d57d58cc9-l8sjd_0fe973cf-559f-4351-86fc-308c879b0541/graceful-termination/0.log"